Sunday, 16 February 2014

Burn srt to avi in Linux

 To split files use avisplit like:
avisplit -s 300 -i video.avi 
 Then to burn subtitle to the video.avi:


 mencoder  in.mp4 -sub mysub.srt -subcp utf8 -subfont-text-scale 2 -sub-bg-color 0 -sub-bg-alpha 100  -subfont-outline 1  -ovc xvid -oac mp3lame -xvidencopts pass=1   -o out.mp4


Note: You need to either import a proper ttf file to your ~/.mplayer OR symlink the folder to a ttf file in /usr/share/fonts. (Doc)



Thursday, 6 February 2014

Install tahoma fonts on Ubuntu

wget -c http://hezardastan.sourceforge.net/persianfonts/tahoma.tar.gz
wget -c http://hezardastan.sourceforge.net/persianfonts/bfonts.tar.gz
sudo mkdir /usr/share/fonts/truetype/ttf-persian-fonts
sudo tar zxvf tahoma.tar.gz -C /usr/share/fonts/truetype/ttf-persian-fonts
sudo tar zxvf bfonts.tar.gz -C /usr/share/fonts/truetype/ttf-persian-fonts
sudo fc-cache -f -v

Tuesday, 4 February 2014

How to counter botnets?

First, what are the indications that you are under attack by botnets?

Usually a very high load on web server without much spike in bandwidth usage or high load on database is a good indication that botnest are involved.

Also, if you use software firewalls like CSF (You should!) you will see on top command that ldf  process eats up a lot of your cpu time. This means a lot of requests are coming through and the firewall needs to deal with them, hence its high cpu usage.

To make sure check your web server access log. Botnets can be recognized by the fact that they send the most number of request  usually to get the the website's root. So you see many requests like:

GET / HTTP/1.1" 200 7288 "http://yourdomain.com/" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.2.3.4) Gecko/20090922 Firefox/3.5.2 (.NET CLR 3.4.5678)"

in your access.log


How to counter botnest?
Don't worry! Botnets are very stupid as they all use the same user agent (at least the one that I've encountered). So it is easy to tame them if the botnet is not very large.

Once you could pinpoint the botnet 'user agent', it is dead simple to counter them: Just add a rule to nginx to deny any request from that particular user agent:


if ($http_user_agent = "200 7288 "http://yourdomain.com/" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.2.3.4) Gecko/20090922 Firefox/3.5.2 (.NET CLR 3.4.5678)") {
  return 444;
}

And of course start nginx. 

That's it. Now the 444 means that nginx returns no information to the client and closes the connection so the botnet effort to bombard your webserver are simply ignored.

However sometimes botnets are so vast that even the above trick does not suffice, as the botnet consumes all the worker connections of nginx. In this case you need to pinpoint the attacking IPs and block them before they reach nginx.

These are scripts that I've written for this purpose:

findbadips.py

#!/usr/bin/env python
import os
from os import system

os.system("netstat -ntu | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -n > data.txt")
#trusted IPs
goodips=['1.2.3.4, '127.0.0.1']
#IPs already blocked in csf.deny
csfdenyips=['1.3.4.2', '5.4.3.2]
#http 444 that should flag bad IPs
flagstrings=['444']
with open('access.log', "r") as f,open('badips.txt', "w") as f2:
    for l in f:
        if (not any(ip in l for ip in goodips) and not any(ip in l for ip in csfdenyips) and any(ip in l for ip in flagstrings)):
              f2.write(l.strip()+'\n')

Then you need to distill the badips.txt to remove repetitions. 

distilips.py

#!/usr/bin/env python
import os
from os import system
import re
import time
#Use awk to distil access log to $IP $TIME and $URL columns
bashcom = "awk '{print $1}' <badips.txt > iprequests.txt"
os.system(bashcom)
#just in case any good ip has sneaked into the list
goodips=['1.2.3.4','127.0.0.1']
distips = []
with open('iprequests.txt', "r") as src,open('distiled-badips.txt', "w") as dest:
    for l in src:
        if (not any(ip in l for ip in goodips)):
   if l not in distips:
distips.append(str(l))
 
    for ip in distips:
dest.write(ip)



After adding to bad ips into csf.deny, and reloading the deny list 'deny -r' you should see that the server load quickly drops to normal.

In case you use cloudflare (you should!) you want to block all the bad IPs at cloulflare level instead of csf.

Thanks to the Cloudflare API, you can do this at one shot:

#!/bin/bash
badIPArray=( 1.2.3.4  4.2.4.2 )
for i in "${badIPArray[@]}"
do
    curl -s https://www.cloudflare.com/api_json.html -d 'a=ban' -d 'tkn=YourAPIKey' -d 'email=you@example.com'  -d 'key=$i';
    echo "posted - $i";
done


You may not see but you can be sure that your adversaries are now pitiful of the money that they spared on petite botnet Sheppard.



Friday, 24 January 2014

mezzanine notes

Error loading MySQLdb module: No module named MySQLdb

On Ubuntu 12.04, to be able to use mysql as backend database in mezzanine, do these Before creating virtualenv do install: 
apt-get install libmysqlclient-dev
apt-get install python-dev
pip install mysql-python

 Then on vritualenv :

 pip install MySQL-python



-----------------------------
decoder jpeg not available

On Ubuntu 12.04 64 bit:
sudo ln -s /usr/lib/x86_64-linux-gnu/libjpeg.so /usr/lib
sudo ln -s /usr/lib/x86_64-linux-gnu/libfreetype.so /usr/lib
sudo ln -s /usr/lib/x86_64-linux-gnu/libz.so /usr/lib
pip install 

Wednesday, 22 January 2014

Fix drupal 6 file permissioins


If you need to fix permissions repeatedly then the following script will help you, it is based on the guidelines given above and performs some checks before any modification to ensure it is not applied on files/directories outside your drupal installation.
#!/bin/bash
if [ $(id -u) != 0 ]; then
        printf "This script must be run as root.\n"
        exit 1
fi
drupal_path=${1%/}
drupal_user=${2}
httpd_group="${3:-www-data}"
# Help menu
print_help() {
cat <<-HELP
This script is used to fix permissions of a Drupal installation
you need to provide the following arguments:
1) Path to your Drupal installation.
2) Username of the user that you want to give files/directories ownership.
3) HTTPD group name (defaults to www-data for Apache).
Usage: (sudo) bash ${0##*/} --drupal_path=PATH --drupal_user=USER --httpd_group=GROUP
Example: (sudo) bash ${0##*/} --drupal_path=/usr/local/apache2/htdocs --drupal_user=john --httpd_group=www-data
HELP
exit 0
}
# Parse Command Line Arguments
while [ $# -gt 0 ]; do
        case "$1" in
                --drupal_path=*)
drupal_path="${1#*=}"
;;
--drupal_user=*)
drupal_user="${1#*=}"
;;
--httpd_group=*)
httpd_group="${1#*=}"
;;
--help) print_help;;
*)
printf "Invalid argument, run --help for valid arguments.\n";
exit 1
esac
shift
done
if [ -z "${drupal_path}" ] || [ ! -d "${drupal_path}/sites" ] || [ ! -f "${drupal_path}/core/modules/system/system.module" ] && [ ! -f "${drupal_path}/modules/system/system.module" ]; then
printf "Please provide a valid Drupal path.\n"
print_help
exit 1
fi
if [ -z "${drupal_user}" ] || [ $(id -un ${drupal_user} 2> /dev/null) != "${drupal_user}" ]; then
printf "Please provide a valid user.\n"
print_help
exit 1
fi
cd $drupal_path
printf "Changing ownership of all contents of "${drupal_path}":\n user => "${drupal_user}" \t group => "${httpd_group}"\n"
chown -R ${drupal_user}:${httpd_group} .
printf "Changing permissions of all directories inside "${drupal_path}" to "rwxr-x---"...\n"
find . -type d -exec chmod u=rwx,g=rx,o= '{}' \;
printf "Changing permissions of all files inside "${drupal_path}" to "rw-r-----"...\n"
find . -type f -exec chmod u=rw,g=r,o= '{}' \;
printf "Changing permissions of "files" directories in "${drupal_path}/sites" to "rwxrwx---"...\n"
cd sites
find . -type d -name files -exec chmod ug=rwx,o= '{}' \;
printf "Changing permissions of all files inside all "files" directories in "${drupal_path}/sites" to "rw-rw----"...\n"
printf "Changing permissions of all directories inside all "files" directories in "${drupal_path}/sites" to "rwxrwx---"...\n"
for x in ./*/files; do
find ${x} -type d -exec chmod ug=rwx,o= '{}' \;
find ${x} -type f -exec chmod ug=rw,o= '{}' \;
done
echo "Done settings proper permissions on files and directories"
Copy the code above to a file, name it "fix-permissions.sh" and run it as follows:
sudo bash fix-permissions.sh --drupal_path=your/drupal/path --drupal_user=your_user_name
Note: The server group name is assumed "www-data", if it differs use the --httpd_group=GROUP argument.

Tuesday, 21 January 2014

Sample text searches for suspicious PHP code

1
grep -Rn "mkdir *(" public_html/
Or
1
grep -RPn "(passthru|shell_exec|system|phpinfo|base64_decode|chmod|mkdir|fopen|fclose|readfile) *\(" public_html/

Tuesday, 14 January 2014

How to move mysql to backend server

This is a great solution to off-load struggling server under load:

On backend server:


nano /etc/network/interfaces

#add following
auto eth1
iface eth1 inet static
address 192.168.1.1
netmask 255.255.255.0
mtu 9000


Don't forget to assign permissions to the front-end user:

GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER ON mydb.* TO 'myuser'@'192.168.1.2' IDENTIFIED BY 'supersecretpassword';
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER ON mydb.* TO 'myuser'@'192.168.1.2.localdomain' IDENTIFIED BY 'supersecretpassword';
FLUSH PRIVILEGES;
quit;

To allow packet comming through:
iptables -A INPUT -i eth1 -s 192.168.1.2 -p tcp --destination-port 3306 -j ACCEPT

service networking restart


On frontend server:

 nano /etc/network/interfaces

#add follwing
auto eth1
iface eth1 inet static
address 192.168.1.2
netmask 255.255.255.0
mtu 9000
Check whether you can connect to the backend:

mysql -u myuser -h 192.168.1.1  -p

service networking restart

 et voila!

thanks:
http://www.cyberciti.biz/tips/how-do-i-enable-remote-access-to-mysql-database-server.html


Saturday, 5 October 2013

Passwordless ssh login

You can login to a remote Linux server without entering password in 3 simple steps using ssky-keygen and ssh-copy-id as explained in this article.

ssh-keygen creates the public and private keys. ssh-copy-id copies the local-host’s public key to the remote-host’s authorized_keys file. ssh-copy-id also assigns proper permission to the remote-host’s home, ~/.ssh, and ~/.ssh/authorized_keys.

This article also explains 3 minor annoyances of using ssh-copy-id and how to use ssh-copy-id along with ssh-agent.

Step 1: Create public and private keys using ssh-key-gen on local-host

jsmith@local-host$ [Note: You are on local-host here]

jsmith@local-host$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/jsmith/.ssh/id_rsa):[Enter key]
Enter passphrase (empty for no passphrase): [Press enter key]
Enter same passphrase again: [Pess enter key]
Your identification has been saved in /home/jsmith/.ssh/id_rsa.
Your public key has been saved in /home/jsmith/.ssh/id_rsa.pub.
The key fingerprint is:
33:b3:fe:af:95:95:18:11:31:d5:de:96:2f:f2:35:f9 jsmith@local-host

Step 2: Copy the public key to remote-host using ssh-copy-id

jsmith@local-host$ ssh-copy-id -i ~/.ssh/id_rsa.pub remote-host
jsmith@remote-host's password:
Now try logging into the machine, with "ssh 'remote-host'", and check in:

.ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.
Note: ssh-copy-id appends the keys to the remote-host’s .ssh/authorized_key.

Step 3: Login to remote-host without entering the password

jsmith@local-host$ ssh remote-host
Last login: Sun Nov 16 17:22:33 2008 from 192.168.1.2
[Note: SSH did not ask for password.]

jsmith@remote-host$ [Note: You are on remote-host here]

The above 3 simple steps should get the job done in most cases.

Note:
Recently I started to get  Agent admitted failure to sign using the key on ubuntu 12.04, but this solved the problem:

# start the ssh-agent in the background
eval "$(ssh-agent -s)

After setting that up refer to "Disable root login" to disable root login for extra security. 

Friday, 4 October 2013

Install CSF proxy on Debian 6

Step 1: Downloading


Config Server Firewall is not currently available in Debian or Ubuntu repositories, and has to be downloaded from the ConfigServer's website.
wget http://www.configserver.com/free/csf.tgz
This will download CSF to your current working directory.

Step 2: Uncompressing


The downloaded file is a compressed from of tar package, and has to be uncompressed and extracted before it can be used.
tar -xzf csf.tgz

Step 3: Installing



Now it is time to execute the CSF's installer script.
cd csf
sh install.sh

The firewall is now installed, but you should check if the required iptables modules are available.
perl /usr/local/csf/bin/csftest.pl

The firewall will work if no fatal errors are reported.

Note: Your IP address was added to the whitelist if possible. In addition, the SSH port has been opened automatically, even if it uses custom port. The firewall was also configured to have testing mode enabled, which means that the iptables rules will be automatically removed five minutes after starting CSF. This should be disabled once you know that your configuration works, and you will not be locked out.

Basic Configuration


CSF can be configured by editing its configuration file csf.conf in /etc/csf:
nano /etc/csf/csf.conf
The changes can be applied with command:
csf -r

Step 1: Configuring ports


The less access there is to your VPS, the more secure your server is. However, not all ports can be closed as the clients must be able to use your services. 

The ports opened by default are the following:
TCP_IN = "20,21,22,25,53,80,110,143,443,465,587,993,995"

TCP_OUT = "20,21,22,25,53,80,110,113,443"

UDP_IN = "20,21,53"

UDP_OUT = "20,21,53,113,123"
Services using the open ports:

  • Port 20: FTP data transfer
  • Port 21: FTP control
  • Port 22: Secure shell (SSH)
  • Port 25: Simple mail transfer protocol (SMTP)
  • Port 53: Domain name system (DNS)
  • Port 80: Hypertext transfer protocol (HTTP)
  • Port 110: Post office protocol v3 (POP3)
  • Port 113: Authentication service/identification protocol
  • Port 123: Network time protocol (NTP)
  • Port 143: Internet message access protocol (IMAP)
  • Port 443: Hypertext transfer protocol over SSL/TLS (HTTPS)
  • Port 465: URL Rendesvous Directory for SSM (Cisco)
  • Port 587: E-mail message submission (SMTP)
  • Port 993: Internet message access protocol over SSL (IMAPS)
  • Port 995: Post office protocol 3 over TLS/SSL (POP3S)
It is possible that you are not using all of these services, so you can close the ports that are not used. I would recommend closing all ports (removing port number form the list), and then adding the ports you need.

Below are port sets that should be opened if you are running the listed service:

On any server:
TCP_IN: 22,53
TCP_OUT: 22,53,80,113,443
UPD_IN: 53
UPD_OUT: 53,113,123
Apache:
TCP_IN: 80,443
FTP server:
TCP_IN: 20,21
TCP_OUT: 20,21
UPD_IN: 20,21
UPD_OUT:20,21
Mail server:
TCP_IN: 25,110,143,587,993,995
TCP_OUT: 25,110
MySQL server (if remote access is required)
TCP_IN: 3306
TCP_OUT: 3306
Note: If you are using IPv6 for your services, you should also configure TCP6_IN, TCP6_OUT, UPD6_IN, and UPD6_OUT similarly to how IPv4 ports were configured earlier.

You can find a comprehensive list of TCP and UDP ports on Wikipedia. You should open the ports of all the services you use.

Step 2: Additional settings


CSF offers a vast number of different options in its configuration files. Some of the most commonly used settings are explained below.

ICMP_IN
Setting ICMP_IN to 1 allows ping to your server and 0 refuses are such requests. If you are hosting any public services, it is recommended to allow ICMP requests, as these can be used to determine whether or not your service is available.

ICMP_IN_LIMIT
Sets the number of ICMP (ping) requests allowed from one IP address within a specified amount of time. There is usually no need to change the default value (1/s)

DENY_IP_LIMIT
Sets the number of blocked IP addresses CSF keeps track of. It is recommended to limit the number of denied IP addresses as having too many blocks may slow down the server performance.

DENY_TEMP_IP_LIMIT
Same as above, but for temporary IP address blocks.

PACKET_FILTER
Filter invalid, unwanted and illegal packets.

SYNFLOOD, SUNFLOOD_RATE and SYNFLOOD_BURST
This offers protection against SYN flood attacks. This slows down the initialization of every connection, so you should enable this only if you know that your server is under attack.

CONNLIMIT
Limits the number of concurrent active connections on port.

Value:
22;5;443;20
would allow 5 concurrent connections on port 22 and 20 concurrent connections on port 443.

PORTFLOOD
Limits the number of connections per time interval that new connections can be made to specific ports. Value:
22;tcp;5;250
would limit block the IP address if more than 5 connections are established on port 22 using TCP protocol within 250 seconds. The block is removed once 250 seconds have passed after the last packet sent by the client to this port. You may add more ports by separating them by commas like described below.
port1;protocol1;connection_count1;time1,port2;protocol2;connection_count2;time2

More settings
CSF offers a wide range of settings which are not covered in this tutorial. The default values are generally good, and can be used on almost any server. The default settings are configured to prevent most flood attacks, port scans and unauthorized access attempts.

If you would, however, like to adjust the configuration in more detail, please read the comments in /etc/csf/csf.conf and edit them as you like. 

Step 3: Applying the Changes


Whenever you are altering the settings in csf.conf, you should save the files and restart CSF in order for the changes to take effect. Once you are ready with the configuration, close the file by pressing Ctrl + X. When you are asked whether to save the changes or not, press Y to save the changes.

After this, you should apply the changes by restarting CSF with command:
csf -r

If everything went like planned, and you are still able to access the server, open the configuration file once more:
nano /etc/csf/csf.conf
and change setting TESTING at the beginning of the configuration file to 0 as shown below:
TESTING = "0"

Save the file, and apply the changes with command:
csf -r

Blocking and Allowing IP Addresses


One of the most basic features of a firewall is the ability to block certain IP addresses. You may deny (blacklist), allow (whitelist) or ignore IP addresses by editing the configuration files csf.deny, csf.allow and csf.ignore.

Blocking IP addresses
If you would like to block an IP address or range, open csf.deny.
nano /etc/csf/csf.deny
Blocked IP addresses or ranges all reserve one line in csf.deny file. If you would like to block IP address 1.2.3.4 as well as IP range 2.3.*.*, you should add the following lines to the file:
1.2.3.4
2.3.0.0/16
IP ranges are represented using the CIDR notation

Allowing IP addresses
If you would like an IP address or range to be excluded from all blocks and filters, you may add them to csf.allow file. Please note that allowed IP addresses are allowed even if they are explicitly blocked in csf.deny file.

Allowing IP addresses works similarly to blocking them. The only difference is that you should edit /etc/csf/csf.allow instead of csf.deny.
nano /etc/csf/csf.allow
Ignoring IP addresses
CSF also offers ability to exclude IP addresses from the firewall filters. IP addresses in csf.ignore will bypass the firewall filters, and can only be blocked if listed in csf.deny file.
nano /etc/csf/csf.ignore
In order to changes take effect, you should restart CSF after editing any of the files described above with command:
csf -r

Find 10 largest files and directories in Linux

To find the largest 10 files (linux/bash):
find . -type f -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
To find the largest 10 directories:
find . -type d -print0 | xargs -0 du | sort -n | tail -10 | cut -f2 | xargs -I{} du -sh {}
Only difference is -type {d:f}.