Wednesday, 27 May 2015

Deploy django on debian 7 with uwsgi and mariadb - complete guide

In local virtualven
=========
pip freeze > requirement.txt

on Server
==============

apt-get install nginx
apt-get install mysql-server

apt-get install python-pip

apt-get install python-virtualenv
apt-get install python-dev

apt-get install memcached
apt-get install python-memcache

========
locale-gen en_US en_US.UTF-8
dpkg-reconfigure locals

==========

apt-get install git-core

=========

adduser bob
su bob

virtualenv ~/.djenv
source ~/.djenv/bin/activate

pip install requirement.txt

pip install python-memcached


===========
CREATE DATABASE mydb;
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER ON mydb.* TO 'myuser'@'localhost' IDENTIFIED BY 'password';
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER ON mydb.* TO 'myuser'@'localhost.localdomain' IDENTIFIED BY 'password';
FLUSH PRIVILEGES;
quit;
==========

To solve
Error loading MySQLdb module: No module named MySQLdb

sudo apt-get install build-essential python-dev

apt-get install libmariadbclient-dev


***Note installing  libmysqlclient-dev causes 'EnvironmentError: mysql_config not found'



pip install MySQL-python

===========================
Activate virtualenv. Then







==============
Error: decoder jpeg not available

apt-get install libjpeg-dev
pip install -I pillow



==========
Oh collation to utf8, otherwise you may get your non-western charachters converted to ?????

  for t in $(mysql --user=root --password=mydbpasswd  --database= mydb -e "show tables";);
    do
       echo "Altering" $t;
       mysql --user=root --password=mydbpasswd --database= mydb -e "ALTER TABLE $t CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;";
    done

Extra (for cache and messaging:

- How to install redis (latest version)

- How to install Celery


=======================================
uwsgi installation (this could be a pain in the backside, but not with this help!)


Basic uwsgi intallation and configuration

Install uwsgi

pip install uwsgi

Basic test

Create a file called test.py:
# test.py
def application(env, start_response):
    start_response('200 OK', [('Content-Type','text/html')])
    return "Hello World"
Run:
uwsgi --http :8000 --wsgi-file test.py
The options mean:
http :8000
use protocol http, port 8000
wsgi-file test.py
load the specified file
This should serve a hello world message directly to the browser on port 8000. Visit:
http://example.com:8000
to check.

Test your Django project

Now we want uwsgi to do the same thing, but to run a Django site instead of the test.py module.
But first, make sure that your project actually works! Now you need to be in your Django project directory.
python manage.py runserver 0.0.0.0:8000
Now run it using uwsgi:
uwsgi --http :8000 --chdir /path/to/your/project --module project.wsgi --virtualenv /path/to/virtualenv
The options mean:
chdir /path/to/your/project
use your Django project directory as a base
module project.wsgi
i.e. the Python wsgi module in your project
virtualenv /path/to/virtualenv
the virtualenv
There is an alternative to using the --module option, by referring instead to the wsgi file:
wsgi-file /path/to/your/project/project/wsgi.py
i.e. the system file path to the wsgi.py file
Point your browser at the server; if the site appears, it means uwsgi can serve your Django application from your virtualenv. Media/static files may not be served properly, but don't worry about that.
Now normally we won't have the browser speaking directly to uwsgi: nginx will be the go-between.

Basic nginx

Install nginx

The version of Nginx from Debian stable is rather old. We'll install from backports.
sudo pico /etc/apt/sources.list     # edit the sources list
Add:
# backports
deb http://backports.debian.org/debian-backports squeeze-backports main
Run:
sudo apt-get -t squeeze-backports install nginx # install nginx
sudo /etc/init.d/nginx start    # start nginx
And now check that the server is serving by visiting it in a web browser on port 80 - you should get a message from nginx: "Welcome to nginx!"

Configure nginx for your site

Check that your nginx has installed a file at /etc/nginx/uwsgi_params. If not, copyhttp://projects.unbit.it/uwsgi/browser/nginx/uwsgi_params to your directory, because nginx will need it. Easiest way to get it:
wget http://projects.unbit.it/uwsgi/export/3fab63fcad3c77e7a2a1cd39ffe0e50336647fd8/nginx/uwsgi_params
Create a file called nginx.conf, and put this in it:
# nginx.conf
upstream django {
    # connect to this socket
    # server unix:///tmp/uwsgi.sock;    # for a file socket
    server 127.0.0.1:8001;      # for a web port socket
    }

server {
    # the port your site will be served on
    listen      8000;
    # the domain name it will serve for
    server_name .example.com;   # substitute your machine's IP address or FQDN
    charset     utf-8;

    #Max upload size
    client_max_body_size 75M;   # adjust to taste

    # Django media
    location /media  {
                alias /path/to/your/project/project/media;      # your Django project's media files
    }

        location /static {
                alias /path/to/your/project/project/static;     # your Django project's static files
        }

    # Finally, send all non-media requests to the Django server.
    location / {
        uwsgi_pass  django;
        include     /etc/nginx/uwsgi_params; # or the uwsgi_params you installed manually
        }
    }
Symlink to this file from /etc/nginx/sites-enabled so nginx can see it:
sudo ln -s ~/path/to/your/project/nginx.conf /etc/nginx/sites-enabled/

Basic nginx test

Restart nginx:
sudo /etc/init.d/nginx restart
Check that media files are being served correctly:
Add an image called media.png to the /path/to/your/project/project/media directory
Visit
If this works, you'll know at least that nginx is serving files correctly.

nginx and uwsgi and test.py

Let's get nginx to speak to the hello world test.py application.
uwsgi --socket :8001 --wsgi-file test.py
This is nearly the same as before, except now we are not using http between uwsgi and nginx, but the (much more efficient) uwsgi protocol, and we're doing it on port 8001. nginx meanwhile will pass what it finds on that port to port 8000. Visit:
to check.
Meanwhile, you can try to have a look at the uswgi output at:
but quite probably, it won't work because your browser speaks http, not uwsgi.

Using sockets instead of ports

It's better to use Unix sockets than ports - there's less overhead.
Edit nginx.conf.
uncomment
server unix:///tmp/uwsgi.sock;
comment out
server 127.0.0.1:8001;
and restart nginx.
Runs uwsgi again:
uwsgi --socket /tmp/uwsgi.sock --wsgi-file test.py
Try http://example.com:8000/ in the browser.

If that doesn't work

Check your nginx error log(/var/log/nginx/error.log). If you see something like:
connect() to unix:///path/to/your/project/uwsgi.sock failed (13: Permission denied)
then probably you need to manage the permissions on the socket (especially if you are using a file not in /tmp as suggested).
Try:
uwsgi --socket /tmp/uwsgi.sock --wsgi-file test.py --chmod-socket=644 # 666 permissions (very permissive)
or:
uwsgi --socket /tmp/uwsgi.sock --wsgi-file test.py --chmod-socket=664 # 664 permissions (more sensible)
You may also have to add your user to nginx's group (probably www-data), or vice-versa, so that nginx can read and write to your socket properly.

Running the Django application with uswgi and nginx

Let's run our Django application:
uwsgi --socket /tmp/uwsgi.sock --chdir /path/to/your/project --module project.wsgi --virtualenv /path/to/virtualenv --chmod-socket=664
Now uwsgi and nginx should be serving up your Django application.

a uwsgi .ini file for our Django application

Deactivate your virtualenv:
deactivate
and install uwsgi system-wide:
sudo pip install uwsgi
We can put the same options that we used with uwsgi into a file, and then ask uwsgi to run with that file:
# django.ini file
[uwsgi]

# master
master                  = true

# maximum number of processes
processes               = 10

# the socket (use the full path to be safe)
socket          = /tmp/uwsgi.sock

# with appropriate permissions - *may* be needed
# chmod-socket    = 664

# the base directory
chdir           = /path/to/your/project

# Django's wsgi file
module          = project.wsgi

# the virtualenv
home            = /path/to/virtualenv

# clear environment on exit
vacuum          = true
And run uswgi using the file:
uwsgi --ini django.ini
Note:
--ini django.ini
use the specified .ini file

Test emperor mode

uwsgi can run in 'emperor' mode. In this mode it keeps an eye on a directory of uwsgi config files, and spawns instances ('vassals') for each one it finds.
Whenever a config file is amended, the emperor will automatically restart the vassal.
# create a directory for the vassals
sudo mkdir /etc/uwsgi
sudo mkdir /etc/uwsgi/vassals
# symlink from the default config directory to your config file
sudo ln -s /path/to/your/project/django.ini /etc/uwsgi/vassals/

# run the emperor as root
sudo uwsgi --emperor /etc/uwsgi/vassals --uid www-data --gid www-data --master
The options mean:
emperor /etc/uwsgi/vassals
look there for vassals (config files)
uid www-data
run as www-data once we've started
gid www-data
run as www-data once we've started
Check the site; it should be running.

Make uwsgi startup when the system boots

The last step is to make it all happen automatically at system startup time.
Edit /etc/rc.local and add:
/usr/local/bin/uwsgi --emperor /etc/uwsgi/vassals --uid www-data --gid www-data --master
before the line "exit 0".
And that should be it!

Source


Tuesday, 26 May 2015

a working my.cnf for Mariadb

#
# The MySQL database server configuration file.
#

[client]
port = 3306
socket = /var/run/mysqld/mysqld.sock

[mysqld_safe]
socket = /var/run/mysqld/mysqld.sock
nice = 0

[mysqld]

user = mysql
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
skip-external-locking
#skip-innodb
default-storage-engine=InnoDB
#
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.
#bind-address = 127.0.0.1

#
# * Fine Tuning
#
key_buffer = 128M
max_allowed_packet = 64M
thread_stack = 192K
thread_cache_size       = 64
# This replaces the startup script and checks MyISAM tables if needed
# the first time they are touched
myisam-recover         = BACKUP
max_connections        = 1000

interactive_timeout= 1
wait_timeout= 1

#log-slow-queries = /var/log/mysql/slow_query.log
#long_query_time = 10
#log_queries_not_using_indexes = 1

table_cache            = 6048
thread_concurrency     = 22

join_buffer_size = 8M
sort_buffer_size = 32M
max_heap_table_size = 4412M
max_connect_errors = 10
tmp_table_size = 4412M




#*** MyISAM Specific options
key_buffer_size = 32M
read_buffer_size = 4M
read_rnd_buffer_size = 8M

myisam_sort_buffer_size = 128M
#myisam_max_sort_file_size = 10G
#myisam_max_extra_sort_file_size = 10G



#finally added
table_definition_cache = 2048


#default-storage-engine = innodb
innodb_buffer_pool_size = 25000M
innodb_log_file_size = 1500M
innodb_flush_method = O_DIRECT #O_DSYNC
innodb_file_per_table = 1
innodb_flush_log_at_trx_commit = 2
innodb_log_buffer_size = 4M
innodb_additional_mem_pool_size = 20M

# num cpu's/cores *2 is a good base line for innodb_thread_concurrency
innodb_thread_concurrency = 16




#
# * Query Cache Configuration
#
query_cache_limit = 2M
query_cache_size  = 256M
query_cache_min_res_unit = 1k

#
# * Logging and Replication
#
# Both location gets rotated by the cronjob.
# Be aware that this log type is a performance killer.
# As of 5.1 you can enable the log at runtime!
#general_log_file        = /var/log/mysql/mysql.log
#general_log             = 1

log_error                = /var/log/mysql/error.log

#server-id = 1
#log_bin = /var/log/mysql/mysql-bin.log
#binlog_do_db = shahbin

#expire_logs_days = 10
#max_binlog_size         = 100M


[mysqldump]
quick
quote-names
max_allowed_packet = 16M

[mysql]
#no-auto-rehash # faster start of mysql but no tab completition

[isamchk]
key_buffer = 16M

!includedir /etc/mysql/conf.d/


Thursday, 22 January 2015

Send email from your own domain using Zoho and Gmail

Gone the days that Google allowed people to send email as contact@mydomain.com using their gmail account for free. Now you need to buy on of Google app products to do so.
But worry not and don't rush to buy a personalized email solution for your domain!  You still can do that with a little help from Zoho and in  this excellent post in which , Dimitrios Savvopoulos explains how to do that flawlessly. 

We will use a combination of Zoho Mail and Gmail.

Before proceeding with the Zoho setup, we have to own or create a Gmail account. When ready, proceed with creating a Zoho account:

 

Zoho setup

At the moment of this writing, Zoho offers free email for up to 10 Users, using a custom domain. Visit the zoho registration page and go for the free option:

Then enter your domain name and your personal details:
The zoho account is almost now complete:
Now click "Proceed to verify domain ownership". To verify your domain I suggest adding a TXT record in your DNS settings. Select TXT Method:
Follow the given instructions to add the TXT record to your dns settings. When ready, click the verify button.
Awesome! Now add a user for your email and skip the groups setting. Then, go to the MX Records page. Click "Proceed to Point MX".
You will now have to enter the following mx records to your dns. Also remove all other existing mx records.
We skip the steps about email migration and mobile access. Click "Proceed to Access ZohoMail".
Now, go to your mail inbox and go to the settings. Then select Email forwarding and POP/IMAP and apply the following settings:
As you can see, we are forwarding all our emails to our gmail account. We also delete the copies from the zoho inbox to prevent exceeding the space limit. Since gmail has a bigger size limit, this would be the first one to reach the limit, but we don't want that.
Then we disable the POP Access and enable the IMAP Access.

 

Gmail settings

We want to use gmail only as a storage medium, therefore we have to configure it to send mail as our domain email.
Open gmail and go to the settings, and then to accounts and import. Select "Add another email address you own" and enter your name and your domain email.
Next, we enter the smtp settings of our zoho account:
And finally, confirm your email address.
If you enabled forwarding from zoho mail already, the confirmation email should be received in your gmail inbox.

Tuesday, 20 January 2015

Simple bash command to convert all tables in a mysql database to utf8

  for t in $(mysql --user=root --password=admin  --database=DBNAME -e "show tables";);
    do 
       echo "Altering" $t;
       mysql --user=root --password=admin --database=DBNAME -e "ALTER TABLE $t CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;";
    done
Source

Friday, 16 January 2015

Install celery with rabbitmq on Django

UPDATE 1 AUGUST 2019 


After a lot of headaches at lost I found a great video  tutorial by Mike Hibbert on how to install rabbitmq with celery and setup Django to use it . It worked like a charm on Django 1.7 1.8 running on Ubuntu . The server OS is Debian 9.

First install celery on machine (no on virtualhost):

root@vps:~# apt-get  install rabbitmq-server


Then define rabitmq user, password and vhost and set permissions:

root@vps:~# rabbitmqctl add_user djuser djpass

root@vps:~# rabbitmqctl add_vhost /djangovhost

root@vps:~# rabbitmqctl set_permissions -p /djangovhost djuser ".*" ".*" ".*"


Restart the rabbitmq server:

root@vps:~# /etc/init.d/rabbitmq-server stop
[ ok ] Stopping message broker: rabbitmq-server.
root@vps:~# /etc/init.d/rabbitmq-server start
[ ok ] Starting message broker: rabbitmq-server.
root@vps:~# 
Install celery on django virtualenv:

(.djenv)root@vps:/root$ pip install celery  

also add these lines to settings.py

#Celery configs
BROKER_HOST = "127.0.0.1"
BROKER_PORT = 5672
BROKER_VHOST = "/djangovhost"
BROKER_USER = "djuser"
BROKER_PASSWORD = "djpass"
Make a celery execution script:

nano /path/to/django/celery.sh

Add this script:

#!/bin/bash

CELERY_BIN="/home/john/.djenv/bin/celery"


# App instance to use
CELERY_APP="myapp"

# Where to chdir at start.
CELERYD_CHDIR="/path/to/django"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=1"

# %n will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n%I.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"

# Workers should run as an unprivileged user.
#   You need to create this user manually (or you can choose
#   a user/group combination that already exists (e.g., nobody).
CELERYD_USER="john"
CELERYD_GROUP="john"

# If enabled pid and log directories will be created if missing,
# and owned by the userid/group configured.
CELERY_CREATE_DIRS=1

export SECRET_KEY="somesecretstring"


----------------

Now you need to deamonize the script using systemd



nano /etc/systemd/system/celery.service

[Unit]
Description=Celery  Daemon
#After=network.target
StartLimitIntervalSec=0
[Service]
Type=simple
User= john
Group= www-data
ExecStart=/path/to/django/celery.sh
#Restart=always
#RestartSec=1
Restart=on-failure
# Configures the time to wait before service is stopped forcefully.
TimeoutStopSec=300
[Install]
WantedBy=multi-user.target


--------

systemctl enable celery

systemctl start celery


check the status

systemctl status celery

The result should be like:

● celery.service - Celery  Daemon
   Loaded: loaded (/etc/systemd/system/celery.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Thu 2019-08-01 22:30:32 CDT; 33min ago
  Process: 28796 ExecStart=/path/to/django/celery.sh (code=exited, status=0/SUCCESS)
 Main PID: 28796 (code=exited, status=0/SUCCESS)



Make sure that the daemon is started

Note: if the damon is not started, you may need to: chomod o+x celery.sh


Now check if celery is actually handling the queue;



celery -A myapp worker -l info



(.djenv)john@server$ celery -A myapp worker -l info

Voila!







Thursday, 2 October 2014

Install tahoma font on Ubuntu

Download tarball that contains all Microsoft fonts

To install them type the followiing in a terminal:

cd ~
wget http://www.stchman.com/tools/MS_fonts/msfonts.tbz
sudo tar -xvjpf ~/msfonts.tbz -C /usr/share/fonts/truetype/msttcorefonts
sudo fc-cache -f -v
rm -f ~/msfonts.tbz




Source

Friday, 15 August 2014

Cacti setup on Debian 7


The only no-nonsense cacti tutorial that actually worked for me on Debian 7. 


SNMP (or Simple Network Management Protocol) is used to gather data on what is going on within a device, such as load, hard disk states, bandwidth. These data are used by network monitoring tools such as Cacti to generate graphs for monitoring purposes.
In a typical deployment of Cacti and SNMP, there will be one or more SNMP-enabled devices, and a separate monitoring server where Cacti collects SNMP feeds from those devices. Please keep in mind that all the devices that need to be monitored must be SNMP enabled. In this tutorial, we will be configuring Cacti and SNMP on the same Linux server for demonstration purpose.

Configure SNMP on Debian or Ubuntu

To install SNMP agent (snmpd) on a Debian-based system, run the following command.
root@server:~# apt-get install snmpd
Then edit its configuration like the following.
root@server:~# vim /etc/snmp/snmpd.conf
# this will make snmpd listen on all interfaces
agentAddress  udp:161

# a read only community 'myCommunity' and the source network is defined
rocommunity myCommunity 172.17.1.0/24

sysLocation    Earth
sysContact     email@domain.tld
After editing the config file, restart snmpd.
root@server:~# service snmpd restart

Configure SNMP on CentOS or RHEL

To install SNMP tools and libraries, run the following command.
root@server:~# sudo yum install net-snmp
Then edit an SNMP config file like the following.
root@server:~# vim /etc/snmp/snmpd.conf
# A user 'myUser' is being defined with the community string 'myCommunity' and source network 172.17.1.0/24
com2sec myUser 172.17.1.0/24 myCommunity

# myUser is added into the group 'myGroup' and the permission of the group is defined
group    myGroup    v1        myUser
group    myGroup    v2c        myUser
view all included .1
access myGroup    ""    any    noauth     exact    all    all    none
root@server:~# service snmpd restart
root@server:~# chkconfig snmpd on
Restart snmpd service, and add it to startup service list.

Testing SNMP

SNMP can be tested by running the snmpwalk command. If SNMP has been successfully configured, this command will generate a ton of output.
root@server:~# snmpwalk -c myCommunity 172.17.1.44 -v1
iso.3.6.1.2.1.1.1.0 = STRING: "Linux mrtg 3.5.0-17-generic #28-Ubuntu SMP Tue Oct 9 19:31:23 UTC 2012 x86_64"
iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.8072.3.2.10
iso.3.6.1.2.1.1.3.0 = Timeticks: (2097) 0:00:20.97

~~ OUTPUT TRUNCATED ~~

iso.3.6.1.2.1.92.1.1.2.0 = Gauge32: 1440
iso.3.6.1.2.1.92.1.2.1.0 = Counter32: 1
iso.3.6.1.2.1.92.1.2.2.0 = Counter32: 0
iso.3.6.1.2.1.92.1.3.1.1.2.7.100.101.102.97.117.108.116.1 = Timeticks: (1) 0:00:00.01
iso.3.6.1.2.1.92.1.3.1.1.3.7.100.101.102.97.117.108.116.1 = Hex-STRING: 07 DD 0B 12 00 39 27 00 2B 06 00 

Configure Cacti with SNMP

In this tutorial, we are setting up both Cacti and SNMP on the same Linux server. So go ahead and install Cacti on your Linux server on which SNMP was just configured.
After installation, Cacti web interface can be accessed using the link "http://172.17.1.44/cacti", of course, in your case, the IP address of your server.
The paths during Cacti installation are usually correct. But they can be double checked if necessary.
During the first-time installation, the default username and password for Cacti are "admin" and "admin". You will be forced to change the password after first login.

Add and Manage Devices to Cacti

Cacti will poll devices based on SNMP string that was configured earlier. In this tutorial, we will add only the local server where SNMP is already enabled.
To add devices, we login as admin and go to console in the Cacti admin panel. We click Console > Devices.
There may already be a device named 'localhost'. We do not need it as we will create fresh graphs. We can delete this device from the list. We add a new device by using the 'add' button.
Next, we set the device parameters.
Now that the device has been added, we specify the graph templates that we want to create. This section can be found in the bottom section of the page.
And then we proceed to creating the graphs.
Here, we create graphs for load average, RAM and hard disk, processor.

Interface Graphs and 64-bit Counters

By default, Cacti uses 32-bit counters in SNMP queries. 32-bit counters are sufficient for most bandwidth graphs, but they do not work correctly for graphs greater than 100 Mbps. If it is known that the bandwidth will exceed more than 100 Mbps, it is always advisable to use 64-bit counters. Using 64-bit counters is not hard at all.
Note: It takes around 15 minutes for Cacti to populate new graphs. There are not alternatives to being patient.

Creating Graph Trees

These snapshots illustrate how to create graph trees and how to add graph to those trees.
We can verify the graph in the graph tree.

User Management

Finally, we create a user with view permission to only the graph that we have created. Cacti has built in user management system, and it is highly customizable.
After completing these steps, we can log in with the user 'user1' and verify that only this user is able to view the graph.
And thus we have deployed a Cacti server in the network monitoring system. Cacti servers are stable, and can deal with tons of graphs without any problems.
Hope this helps.