We provide shared hosting for a huge number of non-profit organizations located at our university. We have been using haproxy and multiple web-nodes for load balancing and reliability ever since our service is used by more and more organizations.
Over the last few weeks we have observed that one of our hosted websites (medienbewusst.de) regularly comes under DoS attacks. This is incredibly annoying and we have now started to investigate this issue in more detail.
All requests are generated by one or multiple-source ip address(es), which always belong to the network of a single company. The attack pattern is almost always identical: First there is a request by a well-known User-Agent (e.g. Firefox 38), which is followed by more than 1500 requests in approximately 3 or 4 minutes, to name one example. These all have the same User-Agent: "Mozilla/4.0 (compatible;)".
::ffff:1.2.3.4 - - [26/Jan/2016:14:47:55 +0100] "GET /fernsehen/20130531/programmtipps-fur-juni-2013.html HTTP/1.1" 200 70439 "-" "Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Firefox/38.0"
::ffff:1.2.3.4 - - [26/Jan/2016:14:47:56 +0100] "GET /2015/08 HTTP/1.1" 200 58606 "-" "Mozilla/4.0 (compatible;)"
::ffff:1.2.3.4 - - [26/Jan/2016:14:47:56 +0100] "GET /cms/favicon.ico HTTP/1.1" 404 63555 "-" "Mozilla/4.0 (compatible;)"
::ffff:1.2.3.4 - - [26/Jan/2016:14:47:56 +0100] "GET /2014/03 HTTP/1.1" 200 57874 "-" "Mozilla/4.0 (compatible;)"
::ffff:1.2.3.4 - - [26/Jan/2016:14:47:56 +0100] "GET /fernsehen/20130531/medienbewusst.de%20& HTTP/1.1" 404 63583 "-" "Mozilla/4.0 (compatible;)"
::ffff:1.2.3.4 - - [26/Jan/2016:14:47:56 +0100] "GET /2014/06 HTTP/1.1" 200 58581 "-" "Mozilla/4.0 (compatible;)"
::ffff:1.2.3.4 - - [26/Jan/2016:14:47:56 +0100] "GET /2015/06 HTTP/1.1" 200 58251 "-" "Mozilla/4.0 (compatible;)"
After talking to a technician, who is responsible for one of these company networks, it turned out that the company uses proxy appliances build by Blue Coat. Their proxy seemingly prefetches the whole webpage really fast, which causes a DoS in association with the used content management system. Users in blogs and forums report that this is typical for Blue Coat proxies and the requests can be identified by the HTTP-Header "HTTP_X_BLUECOAT_VIA" next to the User-Agent.
We now prohibit the requests on our HAProxy-Loadbalancers with the following configuration:
frontend web-1
...
# Block Bluecoat proxy prefetch requests with User-Agent and special HTTP header
acl blocked_ua hdr_sub(user-agent) -i Mozilla/4.0\ (compatible;)
acl blocked_proxy hdr_cnt(X-BlueCoat-Via) gt 0
use_backend blocked-proxy-ua if blocked_ua blocked_proxy
...
backend blocked-proxy-ua
mode http
errorfile 503 /etc/haproxy/errors/503_blocked_proxy_ua.txt
File /etc/haproxy/errors/503_blocked_proxy_ua.txt
HTTP/1.0 503 Service Unavailable
Cache-Control: no-cache
Connection: close
Content-Type: text/html
<html>
<head>
<title>503 - Service Unavailable</title>
</head>
<body>
<h1>Your request was blocked.</h1>
The proxy you are using prefetches the whole webpage with a very high
rate per second. To prevent this DoS attack your request was blocked.
<br />
Please contact your IT Administrator.
</body>
</html>
Sources:
https://www.webmasterworld.com/search_engine_spiders/3749904.htm
http://johannburkard.de/blog/www/spam/Mozilla-4-0-compatible.html