io test of server

1 or 2U Rack space* 5000 GB bandwidth* 2 AMP / 110v power

dd if=/dev/zero of=test bs=64k count=16k conv=fdatasync

A samle taken now:

My Desktop result: 100MB/s
Dedi (under 7 load): 58.6MB/s
UK VPS (no load): 76.9 MB/s
Bursnet VPS: 118 MB/s

Django deployment on Ubuntu 10.04 using nginx and uwsgi

I spend days to figure out how to do this. The problem is that the existing docs are somehow incompelte or too geeky.  So I doccument all the necessary steps that led to a successful deployment, for futre reference. Hopefully this help other beginner uwsgi deployers too!

I used a minimal Ubuntu 10.04 (32bit) so some of the apt-gets mentioned here may not be needed in a normal ubuntu installation.

Prepare the server

apt-get update
apt-get upgrade
apt-get install nano

apt-get install --reinstall language-pack-en

apt-get install libxml2-dev build-essential python-dev python-pip

To import ppa keys you need to:

apt-get install python-software-properties 

Since uwsgi is natively supported from nginx 0.8(?) onward, you need to use the latest nginx package (which is 1.xx) instead of the archaic nginx 0.7x still in debian packages. To install the latest nginx,

sudo su -
echo "deb http://ppa.launchpad.net/nginx/stable/ubuntu lucid main" >> /etc/apt/sources.list
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C300EE8C
apt-get update 
apt-get install nginx

Now install uwsgi package:

add-apt-repository  ppa:uwsgi/release 

apt-get update

apt-get -y install uwsgi-python

Here is how I organize directories:

/ /www
/dje /proj
/static

So let's install virtualenv and make dje (Django environment)

apt-get install python-virtualenv

mkdir /www
cd /www
virtualenv dje
cd /www/dje

source bin/activate 

  pip install django

Then make a new project called  proj

python /www/dje/bin/django-admin.py startproject proj

cd /www/dje/proj

Dont' forget doing usuasl settings.py stuff and syncdb! Then

mkdir static
nano deploy.py

In the the deploy.py paste:

 import os
import sys
from os.path import abspath, dirname, join
from site import addsitedir
sys.path.insert(0, abspath(join(dirname(__file__), "../")))
from django.conf import settings
os.environ["DJANGO_SETTINGS_MODULE"] = "proj.settings"
# sys.path.insert(0, join(settings.PROJECT_ROOT, "apps"))
from django.core.handlers.wsgi import WSGIHandler
application = WSGIHandler()

You are done with the django side, now make uwsgi configs:

nano /etc/uwsgi-python/apps-available/django.xml 

 And paste in it:

<uwsgi>
    <socket>127.0.0.1:4000</socket>
    <pythonpath>/www/dje</pythonpath>
    <app mountpoint="/">
        <script>wsgihandler</script>
    </app>
</uwsgi>

Then, symlink it:

ln -s /etc/uwsgi-python/apps-available/django.xml /etc/uwsgi-python/apps-enabled/django.xml

Finally make the site's nginx config file:

nano /etc/nginx/sites-available/default

And paste in it:

upstream django {

server 127.0.0.1:4000;

}

server {

listen 80;

server_name mysite.com;

location / {

uwsgi_pass django;

include uwsgi_params;

uwsgi_param UWSGI_PYHOME /www/dje;

uwsgi_param UWSGI_SCRIPT deploy; #the name of deploy.py

# uwsgi_param SCRIPT_NAME django;

uwsgi_param UWSGI_CHDIR /www/dje/proj;

}

location ^~ /media/ {

root /www/dje/proj/static;

}

}

That's it just restart nginx and uwsgi and enjoy the combo:

service uwsgi-python restart
service nginx  restart

And to install mysql :

apt-get install mysql-server 

apt-get install python-mysqldb 

special thanks to Jason Wang and Web2py folks. 

How to complately remove php from Ubuntu server?

php_installed=`dpkg -l | grep php| awk '{print $2}' |tr "\n" " "`

# remove all php packge

sudo aptitude purge $php_installed

Pinax deployment with nginx and flup

After A LOT of searches failed attempts, at last I found this solution (kind of) working:

1. first install pinax, using the official guide:

and install the necessary packages

apt-get install python-flup nginx subversion python-mysqldb

2. Modify nginx conf

nano /etc/nginx/nginx.conf

user www-data;
worker_processes  1;

error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
    # multi_accept on;
}

http {
    include       /etc/nginx/mime.types;

    access_log /var/log/nginx/access.log;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;
    tcp_nodelay        on;

    gzip  on;
    gzip_disable "MSIE [1-6]\.(?!.*SV1)";
    gzip_comp_level  6;
    gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

3. define  foo.com for nginx

nano /etc/nginx/sites-available/foo.com

server {
listen 80;
server_name www.foo.com foo.com;

if ($host != 'foo.com') {
rewrite ^/(.*)$ http://foo.com/$1 permanent;
}

access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;

location / {
fastcgi_pass 127.0.0.1:7718;
fastcgi_param PATH_INFO $fastcgi_script_name;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param QUERY_STRING $query_string;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_pass_header Authorization;
fastcgi_intercept_errors off;
fastcgi_param REMOTE_ADDR $remote_addr;
fastcgi_param REMOTE_PORT $remote_port;
fastcgi_param SERVER_ADDR $server_addr;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;
}

location /robots.txt {
alias /www/foo/media/robots.txt;
}


location /site_media/ {
expires 7d;
alias /www/foo/media/;
}

location /media/ {
alias /www/env/lib/python2.6/site-packages/django/contrib/admin/media;
}
}

Boring Symlink：

ln -s /etc/nginx/sites-available/foo.com /etc/nginx/sites-enabled/foo.com

4. Start nginx and fcgi

sudo /etc/init.d/nginx restart
python /www/foo/manage.py runfcgi host=127.0.0.1 port=7718 pidfile=/www/foo/foocom.pid maxspare=2

Some useful commands:

If you don’t know where your Python site directory is:

python -c "from distutils.sysconfig import get_python_lib; print get_python_lib()"

Thanks goes to Thinking Critically

virtualenvwrapper (and install pinax)

First:

 apt-get install python-setuptools

easy_install pip
pip install virtualenv
pip install virtualenvwrapper

Then

nano ~/.bashrc

Now copy the required path

# virtualenvwrapper
export WORKON_HOME=~/.virtualenvs
source /usr/bin/virtualenvwrapper.sh
export PIP_VIRTUALENV_BASE=$WORKON_HOME
export PIP_RESPECT_VIRTUALENV=true

source ~/.bashrc

mkdir ~/.virtualenvs

Usage examples:

mkvirtualenv --no-site-packages myenv

workon myenv

Pinax
Then you can cd to favorite directory and install pinax:

pip install Pinax

And replicate a project:

 pinax-admin setup_project -b social myproj

 Thansk:
http://blog.sidmitra.com/manage-multiple-projects-better-with-virtuale

Mysql database: Migrate and Repair

To repair, just issue this command: 

mysqlcheck -uroot -pxxxxx --auto-repair --optimize --databases your_db

Man page

Migrate mysql using file 

Usually, it is less-time consuming to migrate a database using file, instead of using mysqldump to make a *.sql file, and then restore the database. Sometimes, like when the *.sql file is not available, it is the only way to re-alleviate a website.

So to migrate the file directly:

1)Stop MySQL server; 
2) if necessary, rename the folder containing the individual table files to whatever you want the database to be named (e.g. 'drupal'); 
3) copy the folder directly into /var/lib/mysql; 
4) set permissions on the folder: as root, run 'chown -R mysql /var/lib/mysql/*'; 
5) restart MySQL server; 
6) If you've used a new mysql user, need to update the settings.php accordingly

MYSQLTuner

MySQLTuner  is a perl script for mysql optimization.

To get uset it:



  wget http://mysqltuner.com/mysqltuner.pl
  chmod +x mysqltuner.pl
  ./mysqltuner.pl

I acheived amazing performance boost (server load dropped form 10 to 1) following the suggestions of this script, i.e.:



General recommendations:
    Add skip-innodb to MySQL configuration to disable InnoDB
    Run OPTIMIZE TABLE to defragment tables for better performance
    MySQL started within last 24 hours - recommendations may be inaccurate
    Enable the slow query log to troubleshoot bad queries
    Increase table_cache gradually to avoid file descriptor limits
Variables to adjust:
    query_cache_size (> 64M)
    table_cache (> 64)







Thanks mediakey.dk for introducing this amazing script.



nsd3+nginx+php-fpm+drupal

Install nsd3 and nginx

touch /etc/nsd3/nsd.conf
apt-get install nsd3
apt-get install nginx

Check these to figure out how to set up php5-fpm : (in case the instructions in the first link not works, install fpm from source, as describe in howtoforge )

http://gerardmcgarry.com/blog/how-install-php-fpm-nginx-ubuntu-1004-server
http://www.howtoforge.com/installing-php-5.3-nginx-and-php-fpm-on-ubuntu-debian

add an owner user

mkdir -p /srv/mysite
adduser rootuser
usermod -G www-data rootuser

apt-get install mysql-client mysql-server php5-mysql php5-imagick php5-gd


Create database and set permissions

mysql -u root -p

CREATE DATABASE mysitedb;
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER ON mysitedb.* TO 'mysiteuser'@'localhost' IDENTIFIED BY 'password#';
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER ON mysitedb.* TO 'mysiteuser'@'localhost.localdomain' IDENTIFIED BY 'password';
FLUSH PRIVILEGES;
quit;

Install drupal

cd /srv/mysite
wget http://ftp.drupal.org/files/projects/drupal-6.20.tar.gz
tar zxvf drupal-6.20.tar.gz
mv drupal-6.20/* .
rm -r drupal-6.20 drupal-6.20.tar.gz

set up permissions

cd sites/default/
cp default.settings.php settings.php
chown www-data:www-data settings.php
chmod 775 settings.php
mkdir files
chown www-data:www-data files
chmod 775 files

memcached on drupal 6

environment: ubuntu hardy (8.04), drupal 6.13

memcached

To install memcached, you need to install libevent first:

sudo apt-get install libevent-dev

Install memcached:

mkdir src
cd src
wget http://memcached.googlecode.com/files/memcached-1.4.0.tar.gz
tar xzvf memcached-1.4.0.tar.gz
cd memcached-1.4.0
./configure
make
sudo make install
cd ..

Create control script:

sudo nano /usr/local/bin/memcache.sh

Add the following code:

#!/bin/sh
case "$1" in
start) /usr/local/bin/memcached -d -u root -m 240  -p 11211
;;
stop)  killall memcached
;;
esac

240 is the memory limit for the instance of memcached, the unit is MB.
11211 is the port number.

make it executable :

sudo chmod +x /usr/local/bin/memcache.sh 

start a memcached instance when the server startup:

sudo nano /etc/rc.local

add:

/usr/local/bin/memcache.sh start

start a memcached instance by running:

/usr/local/bin/memcache.sh start

PECL memcache extension for PHP

install php-pear if you have not installed it yet

apt-get install php-pear

install PECL memcache :

pecl install Memcache 

Edit the php.ini

nano/etc/php5/fpm  php.ini file:

add "extension=memcache.so" to it.

Restart nginx and php5-fmp

Memcache API and Integration module

open settings.php of your drupal site ( /sites/default/settings.php ), and add the following to the end of the file :

$conf = array(
   'cache_inc' => './sites/all/modules/memcache/memcache.inc',
 );

note: you may place './sites/all/modules/memcache/memcache.inc' with 'cache_inc' => './sites/all/modules/memcache/memcache.db.inc' to cache data both to memory and database if your memcache's memory limit is small or the memcached instance go offline often. (See the README.txt of Memcache API and Integration for more details. )
download Memcache API and Integration module from http://drupal.org/project/memcache, install and enable it.
Now, the integration of memcached and your drupal site is done. You can view memcache status from /admin/reports/memcache .

source

set up postfix to send mails to google apps

After hours of search and trying several different solution, I found that it is surprisingly simle:

apt-get install postfix

nano /etc/postfix/main.cnf


change the following

mydestination = mydomain.com, localhost.mydomain.com, localhost

to

mydestination = localhost.mydomain.com, localhost

Reboot the server. Done!

Thanks Gyaan Sutra