How to counter botnets?

First, what are the indications that you are under attack by botnets?

Usually a very high load on web server without much spike in bandwidth usage or high load on database is a good indication that botnest are involved.

Also, if you use software firewalls like CSF (You should!) you will see on top command that ldf  process eats up a lot of your cpu time. This means a lot of requests are coming through and the firewall needs to deal with them, hence its high cpu usage.

To make sure check your web server access log. Botnets can be recognized by the fact that they send the most number of request  usually to get the the website's root. So you see many requests like:

GET / HTTP/1.1" 200 7288 "http://yourdomain.com/" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.2.3.4) Gecko/20090922 Firefox/3.5.2 (.NET CLR 3.4.5678)"

in your access.log


How to counter botnest?
Don't worry! Botnets are very stupid as they all use the same user agent (at least the one that I've encountered). So it is easy to tame them if the botnet is not very large.

Once you could pinpoint the botnet 'user agent', it is dead simple to counter them: Just add a rule to nginx to deny any request from that particular user agent:


if ($http_user_agent = "200 7288 "http://yourdomain.com/" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.2.3.4) Gecko/20090922 Firefox/3.5.2 (.NET CLR 3.4.5678)") {
  return 444;
}

And of course start nginx. 

That's it. Now the 444 means that nginx returns no information to the client and closes the connection so the botnet effort to bombard your webserver are simply ignored.

However sometimes botnets are so vast that even the above trick does not suffice, as the botnet consumes all the worker connections of nginx. In this case you need to pinpoint the attacking IPs and block them before they reach nginx.

These are scripts that I've written for this purpose:

findbadips.py

#!/usr/bin/env python
import os
from os import system

os.system("netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n > data.txt")
#trusted IPs
goodips=['1.2.3.4, '127.0.0.1']
#IPs already blocked in csf.deny
csfdenyips=['1.3.4.2', '5.4.3.2]
#http 444 that should flag bad IPs
flagstrings=['444']
with open('access.log', "r") as f,open('badips.txt', "w") as f2:
    for l in f:
        if (not any(ip in l for ip in goodips) and not any(ip in l for ip in csfdenyips) and any(ip in l for ip in flagstrings)):
              f2.write(l.strip()+'\n')

Then you need to distill the badips.txt to remove repetitions. 

distilips.py

#!/usr/bin/env python
import os
from os import system
import re
import time
#Use awk to distil access log to $IP $TIME and $URL columns
bashcom = "awk '{print $1}' <badips.txt > iprequests.txt"
os.system(bashcom)
#just in case any good ip has sneaked into the list
goodips=['1.2.3.4','127.0.0.1']
distips = []
with open('iprequests.txt', "r") as src,open('distiled-badips.txt', "w") as dest:
    for l in src:
        if (not any(ip in l for ip in goodips)):
   if l not in distips:
distips.append(str(l))
 
    for ip in distips:
dest.write(ip)

After adding to bad ips into csf.deny, and reloading the deny list 'deny -r' you should see that the server load quickly drops to normal.

In case you use cloudflare (you should!) you want to block all the bad IPs at cloulflare level instead of csf.

Thanks to the Cloudflare API, you can do this at one shot:

#!/bin/bash
badIPArray=( 1.2.3.4  4.2.4.2 )
for i in "${badIPArray[@]}"
do
    curl -s https://www.cloudflare.com/api_json.html -d 'a=ban' -d 'tkn=YourAPIKey' -d 'email=you@example.com'  -d 'key=$i';
    echo "posted - $i";
done

You may not see but you can be sure that your adversaries are now pitiful of the money that they spared on petite botnet Sheppard.