DDoS from an unexpected direction, or "whether the bots are afraid of the SS?"

Want to tell an interesting story about the interaction with search engines, and in particular with bots great and powerful Yandex.

Preamble. I have a dedicated server, which turns the order of several tens of sites. Never had any problems, machine chipper, new. A few days ago received a report that everything is "frozen". Had to do a remote reboot. Later I noticed that the CPU load increased to 60% from 10% and stayed at that level. Of course, I was suspicious, but you never know.

And yesterday I completely. Reboot — progrus and again calm. All sites go away. After the monitor saw download mysql at 99.9%. Surprised, went to look for the site, which is available to download. Found, weed out other queries and saw that someone persistently hammer 40K requests per second, overloading poor database according to the "most can not". Start digging in the logs and I see the subnet, which are these queries. Connect to the server, prescribed DROP iptables mask /24, and everything is back to normal.

Noticed that ddoser sends the identification of the Yandex bot. Slightly was surprised, but you never know. Decided to check and found that two subnets that I had to ban belongs just after the spiders of Yandex.

After a thorough study why and what the bots were doing on the website with such fury it became clear that the bots have got to the open product filter and spawn on the links incredible bunch of pages that were hard to see with the purpose of indexing. Why not work they have some kind of cutoff for such is not clear. The website with the filter running for more than three years and no problems.

At the moment the filter is accessed from the index via htaccess, bots unban, and I'm waiting for a response from support Yandex.

a Small piece of logs
[29/Oct/2015:13:57:10 ] "GET /catalogue/kotli/?176&filter=1&fldX=0&page=8&page=6&show=10&show=50&page=7&page=5&show=10&page=4&page=6 HTTP/1.1" 200 102 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
87.250.244.29 — - [29/Oct/2015:13:57:10 ] "GET /catalogue/kotli/?176&filter=1&fldX=0&page=8&page=6&show=10&show=50&page=7&page=5&show=10&page=4&page=1 HTTP/1.1" 200 102 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
87.250.244.16 — - [29/Oct/2015:13:57:10 ] "GET /catalogue/kotli/?176&filter=1&fldX=0&page=8&page=6&show=10&show=50&page=7&page=5&show=10&page=4&page=2 HTTP/1.1" 200 102 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
141.8.141.140 — - [29/Oct/2015:13:57:10] "GET /catalogue/kotli/?176&filter=1&fldX=0&page=8&page=6&show=10&page=111&show=50&page=1&page=23&show=30&page=37 HTTP/1.1" 200 102 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
87.250.244.16 — - [29/Oct/2015:13:57:10] "GET /catalogue/kotli/?176&filter=1&fldX=0&page=8&page=6&show=10&page=111&show=50&page=1&page=23&show=30&page=26 HTTP/1.1" 200 102 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
141.8.141.140 — - [29/Oct/2015:13:57:10] "GET /catalogue/kotli/?176&filter=1&fldX=0&page=8&page=6&show=10&page=111&show=50&page=1&page=23&show=30&page=25 HTTP/1.1" 200 102 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
141.8.141.139 — - [29/Oct/2015:13:57:10] "GET /catalogue/kotli/?176&filter=1&fldX=0&page=8&page=6&show=10&page=111&show=50&page=1&page=23&show=30&page=24 HTTP/1.1" 200 102 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
87.250.244.29 — - [29/Oct/2015:13:57:10] "GET /catalogue/kotli/?176&filter=1&fldX=0&page=8&page=6&show=10&page=111&show=50&page=1&page=23&show=30&page=20 HTTP/1.1" 200 102 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
87.250.244.16 — - [29/Oct/2015:13:57:10] "GET /catalogue/kotli/?176&filter=1&fldX=0&page=8&page=6&show=10&page=111&show=50&page=1&page=23&show=30&page=21 HTTP/1.1" 200 102 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
141.8.141.140 — - [29/Oct/2015:13:57:10] "GET /catalogue/kotli/?176&filter=1&fldX=0&page=5&show=10&page=3&show=50&page=1&page=3&page=6&page=23 HTTP/1.1" 200 102 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
141.8.141.139 — - [29/Oct/2015:13:57:10] "GET /catalogue/kotli/?176&filter=1&fldX=0&page=8&page=6&show=10&page=1&page=3&show=30&page=1&page=3&page=37 HTTP/1.1" 200 102 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
87.250.244.29 — - [29/Oct/2015:13:57:10] "GET /catalogue/kotli/?176&filter=1&fldX=0&page=8&page=6&show=10&page=5&page=4&page=1&page=2&page=111 HTTP/1.1" 200 102 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
87.76.13.189 — - [29/Oct/2015:13:57:10] "GET / HTTP/1.1" 200 102 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36"
141.8.141.140 — - [29/Oct/2015:13:57:10] "GET /catalogue/kotli/?176&filter=1&fldX=0&page=8&page=6&show=10&page=8&page=7&page=8&page=6&page=4&page=7 HTTP/1.1" 200 102 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
141.8.141.139 — - [29/Oct/2015:13:57:10] "GET /catalogue/kotli/?176&filter=1&fldX=0&page=8&page=6&show=10&page=8&page=7&page=8&page=6&page=4&page=6 HTTP/1.1" 200 102 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
87.250.244.29 — - [29/Oct/2015:13:57:10] "GET /catalogue/kotli/?176&filter=1&fldX=0&page=8&page=6&show=10&page=8&page=7&page=8&page=6&page=4&page=1 HTTP/1.1" 200 102 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
87.250.244.16 — - [29/Oct/2015:13:57:10] "GET /catalogue/kotli/?176&filter=1&fldX=0&page=8&page=6&show=10&page=8&page=7&page=8&page=6&page=4&page=2 HTTP/1.1" 200 102 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
141.8.141.138 — - [29/Oct/2015:13:57:10] "GET /catalogue/kotli/?176&filter=1&fldX=0&page=8&page=6&show=10&page=4&show=50&show=10&show=30&page=37 HTTP/1.1" 200 102 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
141.8.141.143 — - [29/Oct/2015:13:57:10] "GET /catalogue/kotli/?176&filter=1&fldX=0&page=8&page=6&show=10&page=1&show=50&page=4&show=30&page=37 HTTP/1.1" 200 102 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
141.8.141.137 — - [29/Oct/2015:13:57:10] "GET /catalogue/kotli/?176&filter=1&fldX=0&page=8&page=6&show=10&show=30&page=1&page=3&page=2&page=3&page=1 HTTP/1.1" 200 102 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
141.8.141.132 — - [29/Oct/2015:13:57:10] "GET /catalogue/kotli/?176&filter=1&fldX=0&page=8&page=6&show=10&show=30&page=1&page=3&page=2&page=3&page=6 HTTP/1.1" 200 102 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
87.250.244.15 — - [29/Oct/2015:13:57:10] "GET /catalogue/kotli/?176&filter=1&fldX=0&page=8&page=6&show=10&show=30&page=1&page=3&page=2&page=3&page=4 HTTP/1.1" 200 102 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
87.250.244.38 — - [29/Oct/2015:13:57:10] "GET /catalogue/kotli/?176&filter=1&fldX=0&page=8&page=6&show=10&show=30&page=1&page=3&page=2&page=3&page=5 HTTP/1.1" 200 102 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
87.250.244.14 — - [29/Oct/2015:13:57:10] "GET /catalogue/kotli/?176&filter=1&fldX=0&page=8&page=6&show=10&page=7&page=9&show=30&page=11&show=50 HTTP/1.1" 200 102 "-" "Mozilla/5.0 (compatible; YandexBot/3.0;

And I know if I was prudent — such problems would not exist. But still.

Thank you.
Article based on information from habrahabr.ru

Комментарии

Отправить комментарий

Популярные сообщения из этого блога

Python-digest #8. News, interesting projects, articles and interviews [20 Dec 2013 — 27 Dec 2013]

Изображение
in a Hurry to congratulate all with coming new year. The gift I really is quite small — all the same latest news about Python and surrounding technologies. But there are interesting things about searching, Android applications, graphical interface for databases, OAuth, TDD, CMS and other letter combinations. The results of the vote previous digest with a lead of as many as six votes I not allowed to do the release on 3 January, we expect the extended version of January 10, Thank you owlman for illustration to the issue. By the way, he also writing about Python but unfortunately, not very often. Dreams patch the day to 30 hours instead of 24. the Articles and interviews the the time series Analysis with python An attempt to describe the process of analyzing time series using the statsmodels module. TDD with Django 1.6 An example implementation of the approach code development based on writing tests and subsequent writing code that passes those tests....
Далее...

Performance comparison of hierarchical models, Django and PostgreSQL

Изображение
Good day, dear readers. Today's article will be devoted to the comparison of models of working with hierarchical data in PostgreSQL via Django app. In the article I specifically do not use a clean implementation in the database, because I'm interested in is performance in an environment close to combat. All tests were performed on my data and results are relevant for this data on your data the result may differ from that obtained in the article. Model implementation of the hierarchical structures in the database For working with such structures in PostgreSQL you can use the following modely : the Adjacency model (AM) — model, when the column is stored parent; the Nested Sets (NS) model when paired column stores the range of all sub-elements; the Materialised Path model (MP) model, when the column is stored the full path to the element. Also details about the implementation of hierarchical structures in a relational database, you can read here . To imp...
Далее...

google life search

Изображение
Yesterday Google announced the Live search So, what is it and how to use it? Did it make your search more relevant and convenient for users? How to get it, where to enter a query to see live results? The algorithm is still driven around and while fully works only for English queries. You enter the query and see the results of the search in the middle of the page in the section Latest result for “your request”. Can try enter your query here . For example, a request Obama. About the middle of the page is visible to block the Latest results that are updated in real-time. New results on request Obama constantly updated and added to the search results. If this text is annoying, you can pause by clicking "Pause", but completely remove the "live search" from the results it is impossible. Adam Smith, a Google employee, guiding the development of new search system in real time, says that not all queries can be seen regularly updated information: ...
Далее...