HTTP DDoS: “Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)”

Our servers were attacked recently by a constant stream of HTTP POST requests.  The requests were coming in from a large range of IP addresses, at a rate of about 5-10 per second, with random POST data.  However, all the requests had the same UserAgent, they always accessed the non-www form of the same domain, and were all POSTs to the root (/) URL of the site only.

Good descriptions of the attack can be found here and here (see Scott_G’s post).  It proved to be particularly difficult to deal with.

A quick script to read the logs confirmed that there were many thousands of IP addresses involved, with very few duplicates.  At this point it wasn’t clear if the IP addresses were being spoofed, which would be consistent with the large variety of addresses involved, and the relatively slow rate of attack would be consistent with a smaller number of hosts than what appears.  In any case, it didn’t seem practical to attempt to block the requests by IP address.

I first tried Apache <Limit> and RewriteRule directives to block matching requests:

<Location "/">
  <Limit POST>
    order allow,deny
    deny from all
  </Limit>
</Location>
<Location ~ "^/.+/">
  <Limit POST>
    order allow,deny
    allow from all
  </Limit>
</Location>

or

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} "^Mozilla/4\.0 \(compatible; MSIE 6\.0; Windows NT 5\.1; SV1\)$"
RewriteCond %{REQUEST_METHOD} =POST
RewriteRule ^/$ http://127.0.0.1/ [R=301,L]

This reduced the load of each request so they were completing faster, but requests were coming in at an increasing rate, which was enough to tie up all the web server threads, essentially starving the web servers of connections.  On the whole, the web servers were performing well, but it was clear that sites hosted on the servers were starting to experience noticeable connection delays.

I did wonder why such a large botnet would bother targeting a rather insignificant site.  It was not a hack attempt, as the requests were fairly uniform and made no exploit attempts.  And it was not a DDoS intended to take down the server, which it could easily have done with the large IP farm it apparently commands.  The cause may be a very large botnet using traffic to random domains as a subversion tactic to throw off anyone looking for their command & control.  As the botnet is just generating random traffic for obfuscation, it’s not concerned with the 403s or 301s it was getting in response.  Unlike Slowloris, R.U.D.Y., and other slow HTTP attacks, the resource starvation here is only a side-effect and (thankfully) not the real intention.  Requests were actually completing in a timely manner.

Next I tried iptables to stop the traffic before it reaches the web server using packet inspection with the string module:

iptables -N ddos
iptables -A ddos -m string --string ! 'Host: domainname.com' --algo bm -j RETURN
iptables -A ddos -m string --string ! 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)' --algo bm -j RETURN
iptables -A ddos -m string --string ! 'POST / HTTP' --algo bm -j RETURN
iptables -A ddos -j DROP
iptables -I INPUT -p tcp --dport 80 -j ddos

This didn’t work either because packet inspection happens too late – the packet containing the desired payload is not the first one in a request message, and by the time it gets looked at, enough packets are already received to establish connection and tie up the web servers.  In fact, dropping packets made things worse as this attack was not meant to take down the server at all, and dropping packets actually had the effect of a R.U.D.Y. attack, tying up the web server threads much much longer waiting for those missing packets to arrive.  I had to abandon this approach entirely.

Inspection of the incoming packets using iptables did however confirm that the IP addresses were not spoofed, as the traffic was well formed, requiring two-way communication, and there was a wide variety of TTLs, also supporting this conclusion.

The one good piece of news was that the botnet respects DNS.  The fact that it was targeting the servers for the one particular domain hinted at this, and solutions used by others facing this attack also indicated that switching hosts didn’t help – the traffic moves with the site.

Based on this, I moved the non-www form of the domain to another host, so that the other sites on the servers are no longer affected.  On the new host, I set up the RewriteRule to drop the traffic – necessary because apparently the botnet respects redirects as well! – and redirected all other traffic back to the original servers.  This solution was not so satisfying as it just offset the traffic to some poor shared hosting server to deal with, but it got it off my back.

Other solutions to consider include installing Nginx as a proxy in front of Apache, as it can easily be configured to drop the connections without tying up web server threads and has a much higher capacity.  Other firewall/proxy solutions on independent hardware are also possible – my solution is essentially a form of this.