How to set up a Linode environment for the Pyramid Framework 1.5, Python 3, PostgreSQL 9.1 on Debian 7.5

I’m in the process of working on a new startup, and I finally decided to break down and do things right this time, which means leaving PHP and my custom written framework in the past. I’m not really a system administrator, so I figured when I set up my Linode there would be a guide out there to go by. To my surprise, even the Linode hosted config repository didn’t have what I needed or it was out of date. So, I just decided to do it all myself and learn it from scratch.  A couple caveats:

1. Like I said, I’m not a system administrator, so there are probably either mistakes or inaccuracies below. I very much appreciate it if you dropped me a line to let me know if there are better ways to do things or if something that I’ve done is now out of date. This is also a work in progress, so some sections may be incomplete.
2. This was a big project of trial and error, so I’ve documented my best guess as to the proper order of things. But often I had to start over or I missed things that I had to add back in later.
3. I’m assuming your computer is running Windows
4. Hashmarks (#) are comments, and you should ignore everything after them
5. Words in all capital letters should be replaced with your own values

Step 1 – Get a linode VPS

Go here and grab a cheap plan for development. The great thing about linode is that you can bump it up to a larger plan when you need to. After you’ve gotten your account set up, log into your manager and set up a Debian 7.5 profile and disk image, and then boot your server.

Step 2 – Connect to your new server and set up the basics

You are going to need a terminal program. I like putty. So grab it, open it up, and then use the root password in your welcome email to connect to your server using the IP address and a secure connection (SSH, port 22) Once you’re logged in as root, you’ll get a command prompt, and we’ll start setting things up.

Set the hostname:

echo "YOURHOST" > /etc/hostname
hostname -F /etc/hostname
vim /etc/hosts


Now alter the file with your new hostname

127.0.1.1       YOURHOST


Add another user besides root:

adduser YOURUSER
usermod -a -G sudo YOURUSER  #Allow the user to 'sudo' (act as root)
usermod -a -G www-data YOURUSER #Put this new user into the web group


Set the time zone
dpkg-reconfigure tzdata

And alter it to reflect your desired time zone (I always use utc, ‘Europe/Dublin’)
Current default time zone: 'Europe/Dublin' Local time is now: Sun Jun 15 19:27:43 IST 2014. Universal Time is now: Sun Jun 15 18:27:43 UTC 2014.

Now log out of root, and reconnect with your new user. We will start using sudo.

Set 30 min ssh timeout:
sudo vim /etc/ssh/sshd_config

Change the following settings:

ClientAliveInterval 600
ClientAliveCountMax 3


Set up your web directory with the proper permissions (from here). This is optional since we will be using mod-wsgi later to serve our Python scripts. I’ll let you read the link for explanations.

sudo chown -R www-data:www-data /var/www
sudo chmod go-rwx /var/www
sudo chmod go+x /var/www
sudo chgrp -R www-data /var/www
sudo chmod -R go-rwx /var/www
sudo chmod -R g+rx /var/www
sudo chmod -R g+rwx /var/www
sudo chown :www-data /var/www
sudo chmod g+s /var/www


Step 3 – Let’s set up apache and python tools

I’ve stolen most of this from a digital ocean tutorial and Linode documentation

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install apache2


Now,we need to set this max clients setting TEMPORARILY, for our low memory test VPS
sudo vim /etc/apache2/apache2.conf

Alter the file so that:
maxclients=20

Now let’s get python tools installed. I don’t know what exactly setuptools is, but you’re going to need it. Virtualenv is very handy, but it’s very hard to wrap your head around how it works. Basically, Python is capable of keeping all of its installed libraries and applications inside of a ‘box’ called a virtualenv.  It’s very handy, because you can have one version of Python with various extensions running on your system (probably 2.7), and you can have a different version of Python just to run your application (in this case version 3). You ‘activate’ the environment, and the system points all of the environment variables to your ‘box’… Then when you’re done you ‘deactivate’ and all of the environment variables get returned to their original state.

sudo apt-get install python-setuptools
sudo apt-get install python-virtualenv


Now we set up the virtualenv (I’m calling my app ‘YOURPROJ’, just because that’s what I’m working on)

cd /home/YOURUSER
mkdir YOURPROJ #what I am going to call my project
cd YOURPROJ
sudo virtualenv -p /usr/bin/python3 YOURPROJ-env #Create it
source /home/YOURUSER/YOURPROJ/YOURPROJ-env/bin/activate #Activate it


Step 4 – Install all of your needed applications inside the virtualenv

sudo apt-get install python3

PostgreSQL; more detail here

sudo apt-get install postgresql-9.1
sudo apt-get build-dep python-psycopg2
sudo apt-get install python-psycopg2 (needed?)
sudo service postgresql restart



sudo apt-get install libapache2-mod-wsgi-py3
sudo a2enmod mod-wsgi
sudo apt-get install python3-pip #lets you download python packages
sudo /home/YOURUSER/YOURPROJ/YOURPROJ-env/bin/pip-3.2 install psycopg2
sudo /home/YOURUSER/YOURPROJ/YOURPROJ-env/bin/pip-3.2 install pyramid #this is our framework


You’ll notice that I’ve started using full paths. Because we are installing Python 3 on a system that has Python 2.7, we’ve got to be very careful to reference the specific Python 3 installation so that all of our dependencies get worked out. The following packages are ones that I use for my applications. You may not need them:

pip install requests #emails
pip install arrow #dates
pip install pyramid_debugtoolbar #Pyramid's debug toolbar
pip install html2textas



deactivate #close our virtualenv


And now we can export the config file for the project we just created in case we ever need to know what we’ve installed.
sudo /home/YOURUSER/YOURPROJ/YOURPROJ-env/bin/pip-3.2 freeze > requirements.txt

Now, we are also going to use a package called fabric, which basically allows us to write scripts for deployment of your website. The problem is that fabric doesn’t work on python 3. So what we’re going to do is create another virtualenv just for fabric

sudo virtualenv --no-site-packages -p /usr/bin/python fabric-env #The --no-site-packages ensures that this is started up completely clean
source /home/YOURUSER/YOURPROJ/fabric-env/bin/activate
sudo /home/YOURUSER/YOURPROJ/fabric-env/bin/pip install fabric #no python3 support
deactivate


Here’s another thing that sucks. We’re going to need this package called markupsafe. It’s not compatible with python 3.2, but I did find out that we can downgrade it to a version that doesn’t have the incompatibility. Note, I’m pretty sure that this needs to be done inside the virtualenv.

source /home/YOURUSER/YOURPROJ/YOURPROJ-env/bin/activate
sudo /home/YOURUSER/YOURPROJ/YOURPROJ-env/bin/pip-3.2 uninstall markupsafe
sudo /home/YOURUSER/YOURPROJ/YOURPROJ-env/bin/pip-3.2 install markupsafe==0.15
deactivate


Step 5 – Finish setting up Apache

More info here sudo vim /etc/apache2/sites-enabled/000-default Alter the file so that the following settings look like below, except, of course, use your project name instead of mine

WSGIDaemonProcess YOURPROJ python-path=/home/YOURUSER/YOURPROJ:/home/YOURUSER/YOURPROJ/YOURPROJ-env/lib/python3.2/site-packages
WSGIProcessGroup YOURPROJ
WSGIScriptAlias / /home/YOURUSER/YOURPROJ/YOURPROJ_proj/wsgi.py


sudo service apache2 restart

Step 6 – Your firewall

We’re going to do a quick config on the firewall that comes with Debian, called iptables. sudo vim /etc/iptables.firewall.rules Copy the following config into the file. Note that you will have to replace the xxx.xxx on the postgres line with your IP address or network if you want to connect to your database remotely:

*filter

COMMIT



sudo iptables-restore < /etc/iptables.firewall.rules
sudo iptables -L sudo vim /etc/network/if-pre-up.d/firewall


Now put the following text into the file;

#!/bin/sh /sbin/iptables-restore < /etc/iptables.firewall.rules


sudo chmod +x /etc/network/if-pre-up.d/firewall

Here are a couple useful lines that you don’t need to run now, but remembering case you need to check the firewall log:

iptables logging config
sudo vim /etc/rsyslog.conf

how to see iptables log
sudo tail -5000 /var/log/kern.log | grep DPT=80

Step 7 – Set up your database (PostgreSQL)

More info here, here, and here

First, we will use the psql tool to make a database user

sudo -u postgres psql template1
ALTER USER postgres with encrypted password 'YOURPASSWORDMD5HASH'; #You will need to md5 your password and put the hash here. You can find a website that will give you an md5 pretty easily
q


And add a working user and a database for your project

psql -U YOURDBUSER template1 CREATE DATABASE YOURPROJ;
q


Now, we need to edit the PostgreSQL configuration file to reflect the new user.
sudo vim /etc/postgresql/9.1/main/pg_hba.conf

local all postgres md5

sudo createuser -U postgres -d -e -E -l -P -r -s YOURPROJ

And now we edit the same configuration file to allow remote connections. Please be aware that this configuration allows anyone to connect to the database, which is not okay for production environments. If you want to restrict access, you’ll need to put in the IP address of your network that you’ll be working from.
sudo vim /etc/postgresql/9.1/main/pg_hba.conf

local all all md5 host all all 0.0.0.0/0 md5

Now edit the other PostgreSQL configuration file to tell it to listen for connections
sudo vim /etc/postgresql/9.1/main/postgresql.conf

listen_addresses = '*'

sudo /etc/init.d/postgresql restart

Step 8 – Source control with git and github

Go ahead and set up a Github account which will be our origin (main) repository. Now, tell git to put your project under source control, but we want to ignore our two virtualenvs and also compiled Python files (.pyc)
cd /home/YOURUSER/YOURPROJ
git init
sudo vim /home/YOURUSER/YOURPROJ/.gitignore

Enter this into the file:

YOURPROJ-env
fabric-env
*.pyc



git add .
git commit -a -m 'Initial commit of this project'


Now, we tell it that we want to use our Github account as our origin repository

git remote add origin https://github.com/YOURGITACCOUNT/YOURGITPROJ.git


Now that it knows about this origin, we are going to ‘push’ the commit that we just made locally above to Github
git push -u origin master

Step 9 – Create our pyramid project package


sudo /home/YOURUSER/YOURPROJ-env/bin/pcreate -s alchemy YOURPACKAGE #Whatever you want to call your initial package inside your project
sudo /home/YOURUSER/YOURPROJ/YOURPROJ-env/bin/python3 setup.py develop
sudo /home/YOURUSER/YOURPROJ/YOURPROJ-env/bin/python3 setup.py test -q


Edit the development.ini file for this new package. Note, that I am not going to put the database connection string in source control, so we are inserting a dummy connection string right now
vim /home/YOURUSER/YOURPROJ/YOURPACKAGE/development.ini

Set port to 8080
sqlalchemy.url = postgresql://YOURDBUSER:xxxxxx@localhost:5432/YOURPROJ

and while we are in this file, if you want the development toolbar, you’ll need to enter the IP address that you’ll be working from so the toolbar will show up:
debugtoolbar.hosts = xxxxxxxxxx #put in your IP address

Since we are not putting the database connection string inside of source control, will need to create a new file that will hold it
vim database.ini #in same directory as development.ini


[postgres]
connstring = postgresql://YOURDBUSER:xxxxxxxxxxxx@localhost:5432/YOURPROJ #You will put your real connection string here


We will need to go back to our .gitignore file and tell git to keep this new file out of the repository. We are also going to put in a database2.ini entry for our test database
sudo vim /home/YOURUSER/YOURPROJ/.gitignore


YOURPROJ-env
fabric-env
*.pyc
database.ini
database2.ini


When our project is started, we will need to enter some code to make the system load our actual database connection string and overwrite the dummy value. This is going to happen in the __init__.py file one directory down from development.ini. Imports at the top:

from configparser import SafeConfigParser import os


Now right under the def main(global_config, **settings):

parser = SafeConfigParser()
iniloc = os.path.abspath(os.path.join(os.path.dirname( __file__ ), '..', 'database.ini'))
read_list = parser.read(iniloc)
connstring = parser.get('postgres', 'connstring')
settings['sqlalchemy.url'] = connstring


Step 10 – Deployment (work in progress)

We are going to use this tool ‘fabric’ to script our deployment process. More info here. 

So we open up the fabric script file
sudo vim /home/YOURUSER/YOURPROJ/fabfile.py

and put our deployment script in there.

For now, I’m just going to post it as is, but I will come back and explain it later. For this to work, you have to finish the database backup section below with the latest directory modifications.

 
from fabric.api import local
from fabric.context_managers import lcd

def backup_project(backup_date):
    local('sudo cp -R /home/YOURUSER/YOURPROJ/ /home/YOURUSER/backups/backup' + backup_date)

def new_branch(branch_name, db):
    with lcd('/home/YOURUSER/dev'):
        #local('mkdir ' + branch_name)
        local('git clone https://github.com/YOURGITACCOUNT/YOURGITPROJ ' + branch_name) 
    with lcd('/home/YOURUSER/dev/' + branch_name):
        local('sudo cp /home/YOURUSER/YOURPROJ/YOURPACKAGE/database' + db + '.ini ./YOURPACKAGE/database.ini')

def refresh_test_db(db_name):
    if db_name != 'YOURPROJ':
        with settings(user="YOURUSER"):
            if local('psql -l -U postgres | grep ' + db_name + ' | wc -l') == 1:
                local('dropdb ' + db_name + ' -U postgres')
            local('createdb -T template0 ' + db_name + ' -U postgres')
            local('gunzip -c /home/YOURUSER/backups/database/latest/YOURPROJ.sql.gz > /home/YOURUSER/backups/database/latest/YOURPROJ.sql')
            local('psql -U postgres -d ' + db_name + ' -f /home/YOURUSER/backups/database/latest/YOURPROJ.sql')
            local('rm /home/YOURUSER/backups/database/latest/YOURPROJ.sql')


def prepare_deployment(branch_name):
    local('/home/YOURUSER/YOURPROJ/YOURPROJ-env/bin/python3 /var/www/YOURPROJ/YOURPACKAGE/setup.py test -q')
    local('git add -p && git commit')
    local('git checkout master && git merge ' + branch_name)

def deploy(branch_name):
    with lcd('/home/YOURUSER/YOURPROJ'):
        local('git pull /home/YOURUSER/dev/' + branch_name)
        #local('sudo service apache2 restart')

You will run these scripts like this:

/home/YOURUSER/YOURPROJ/fabric-env/bin/fab -f /home/YOURUSER/YOURPROJ/fabfile.py backup_project:date
/home/YOURUSER/YOURPROJ/fabric-env/bin/fab -f /home/YOURUSER/YOURPROJ/fabfile.py new_branch:test,2
/home/YOURUSER/YOURPROJ/fabric-env/bin/fab -f /home/YOURUSER/YOURPROJ/fabfile.py refresh_test_db:YOURTESTDBNAME
/home/YOURUSER/YOURPROJ/fabric-env/bin/fab -f /home/YOURUSER/YOURPROJ/fabfile.py prepare_deployment:test
/home/YOURUSER/YOURPROJ/fabric-env/bin/fab -f /home/YOURUSER/YOURPROJ/fabfile.py deploy:test


Step 11 – Database backups

Set up database backups with pg_backup.sh. More info here Edit pg_backup.config #Slight modifications to the script to allow for a latest directory. Keep in mind that you’ve got a make the modifications in pg_backup for this to include a latest directory.

mkdir /home/YOURUSER/backups/database/latest


Use pgpass to allow automatic script execution. More info here . make sure you are logged in as YOURUSER.

cd ~
touch .pgpass
vim .pgpass


*:*:*:YOURDBUSER:YOURDBPASS



chmod 0600 ~/.pgpass



bash /home/YOURUSER/YOURPROJ/YOURPACKAGE/YOURPACKAGE/scripts/pg_backup.sh


Run regularly with Cron
crontab -e 0 3 * * * /home/YOURUSER/YOURPROJ/YOURPACKAGE/YOURPACKAGE/scripts/pg_backup.sh > /dev/null 2>&1

check status
/etc/init.d/cron status

check the log
sudo grep CRON /var/log/syslog