I’m in the process of working on a new startup, and I finally decided to break down and do things right this time, which means leaving PHP and my custom written framework in the past. I’m not really a system administrator, so I figured when I set up my Linode there would be a guide out there to go by. To my surprise, even the Linode hosted config repository didn’t have what I needed or it was out of date. So, I just decided to do it all myself and learn it from scratch. A couple caveats:
- Like I said, I’m not a system administrator, so there are probably either mistakes or inaccuracies below. I very much appreciate it if you dropped me a line to let me know if there are better ways to do things or if something that I’ve done is now out of date. This is also a work in progress, so some sections may be incomplete.
- This was a big project of trial and error, so I’ve documented my best guess as to the proper order of things. But often I had to start over or I missed things that I had to add back in later.
- I’m assuming your computer is running Windows
- Hashmarks (#) are comments, and you should ignore everything after them
- Words in all capital letters should be replaced with your own values
Step 1 – Get a linode VPS
Go here and grab a cheap plan for development. The great thing about linode is that you can bump it up to a larger plan when you need to. After you’ve gotten your account set up, log into your manager and set up a Debian 7.5 profile and disk image, and then boot your server.
Step 2 – Connect to your new server and set up the basics
You are going to need a terminal program. I like putty. So grab it, open it up, and then use the root password in your welcome email to connect to your server using the IP address and a secure connection (SSH, port 22) Once you’re logged in as root, you’ll get a command prompt, and we’ll start setting things up.
Set the hostname:
echo "YOURHOST" > /etc/hostname
hostname -F /etc/hostname
vim /etc/hosts
Now alter the file with your new hostname
127.0.1.1 YOURHOST
Add another user besides root:
adduser YOURUSER
usermod -a -G sudo YOURUSER #Allow the user to 'sudo' (act as root)
usermod -a -G www-data YOURUSER #Put this new user into the web group
Set the time zone
dpkg-reconfigure tzdata
And alter it to reflect your desired time zone (I always use utc, ‘Europe/Dublin’)
Current default time zone: 'Europe/Dublin' Local time is now: Sun Jun 15 19:27:43 IST 2014. Universal Time is now: Sun Jun 15 18:27:43 UTC 2014.
Now log out of root, and reconnect with your new user. We will start using sudo.
Set 30 min ssh timeout:
sudo vim /etc/ssh/sshd_config
Change the following settings:
ClientAliveInterval 600
ClientAliveCountMax 3
Set up your web directory with the proper permissions (from here). This is optional since we will be using mod-wsgi later to serve our Python scripts. I’ll let you read the link for explanations.
sudo chown -R www-data:www-data /var/www
sudo chmod go-rwx /var/www
sudo chmod go+x /var/www
sudo chgrp -R www-data /var/www
sudo chmod -R go-rwx /var/www
sudo chmod -R g+rx /var/www
sudo chmod -R g+rwx /var/www
sudo chown :www-data /var/www
sudo chmod g+s /var/www
Step 3 – Let’s set up apache and python tools
I’ve stolen most of this from a digital ocean tutorial and Linode documentation
sudo apt-get update
sudo apt-get upgrade
sudo apt-get install apache2
Now,we need to set this max clients setting TEMPORARILY, for our low memory test VPS
sudo vim /etc/apache2/apache2.conf
Alter the file so that:
maxclients=20
Now let’s get python tools installed. I don’t know what exactly setuptools is, but you’re going to need it. Virtualenv is very handy, but it’s very hard to wrap your head around how it works. Basically, Python is capable of keeping all of its installed libraries and applications inside of a ‘box’ called a virtualenv. It’s very handy, because you can have one version of Python with various extensions running on your system (probably 2.7), and you can have a different version of Python just to run your application (in this case version 3). You ‘activate’ the environment, and the system points all of the environment variables to your ‘box’… Then when you’re done you ‘deactivate’ and all of the environment variables get returned to their original state.
sudo apt-get install python-setuptools
sudo apt-get install python-virtualenv
Now we set up the virtualenv (I’m calling my app ‘YOURPROJ’, just because that’s what I’m working on)
cd /home/YOURUSER
mkdir YOURPROJ #what I am going to call my project
cd YOURPROJ
sudo virtualenv -p /usr/bin/python3 YOURPROJ-env #Create it
source /home/YOURUSER/YOURPROJ/YOURPROJ-env/bin/activate #Activate it
Step 4 – Install all of your needed applications inside the virtualenv
sudo apt-get install python3
PostgreSQL; more detail here
sudo apt-get install postgresql-9.1
sudo apt-get build-dep python-psycopg2
sudo apt-get install python-psycopg2 (needed?)
sudo service postgresql restart
sudo apt-get install libapache2-mod-wsgi-py3
sudo a2enmod mod-wsgi
sudo apt-get install python3-pip #lets you download python packages
sudo /home/YOURUSER/YOURPROJ/YOURPROJ-env/bin/pip-3.2 install psycopg2
sudo /home/YOURUSER/YOURPROJ/YOURPROJ-env/bin/pip-3.2 install pyramid #this is our framework
You’ll notice that I’ve started using full paths. Because we are installing Python 3 on a system that has Python 2.7, we’ve got to be very careful to reference the specific Python 3 installation so that all of our dependencies get worked out. The following packages are ones that I use for my applications. You may not need them:
pip install requests #emails
pip install arrow #dates
pip install pyramid_debugtoolbar #Pyramid's debug toolbar
pip install html2textas
deactivate #close our virtualenv
And now we can export the config file for the project we just created in case we ever need to know what we’ve installed.
sudo /home/YOURUSER/YOURPROJ/YOURPROJ-env/bin/pip-3.2 freeze > requirements.txt
Now, we are also going to use a package called fabric, which basically allows us to write scripts for deployment of your website. The problem is that fabric doesn’t work on python 3. So what we’re going to do is create another virtualenv just for fabric
sudo virtualenv --no-site-packages -p /usr/bin/python fabric-env #The --no-site-packages ensures that this is started up completely clean
source /home/YOURUSER/YOURPROJ/fabric-env/bin/activate
sudo /home/YOURUSER/YOURPROJ/fabric-env/bin/pip install fabric #no python3 support
deactivate
Here’s another thing that sucks. We’re going to need this package called markupsafe. It’s not compatible with python 3.2, but I did find out that we can downgrade it to a version that doesn’t have the incompatibility. Note, I’m pretty sure that this needs to be done inside the virtualenv.
source /home/YOURUSER/YOURPROJ/YOURPROJ-env/bin/activate
sudo /home/YOURUSER/YOURPROJ/YOURPROJ-env/bin/pip-3.2 uninstall markupsafe
sudo /home/YOURUSER/YOURPROJ/YOURPROJ-env/bin/pip-3.2 install markupsafe==0.15
deactivate
Step 5 – Finish setting up Apache
More info here sudo vim /etc/apache2/sites-enabled/000-default
Alter the file so that the following settings look like below, except, of course, use your project name instead of mine
WSGIDaemonProcess YOURPROJ python-path=/home/YOURUSER/YOURPROJ:/home/YOURUSER/YOURPROJ/YOURPROJ-env/lib/python3.2/site-packages
WSGIProcessGroup YOURPROJ
WSGIScriptAlias / /home/YOURUSER/YOURPROJ/YOURPROJ_proj/wsgi.py
sudo service apache2 restart
Step 6 – Your firewall
We’re going to do a quick config on the firewall that comes with Debian, called iptables. sudo vim /etc/iptables.firewall.rules
Copy the following config into the file. Note that you will have to replace the xxx.xxx on the postgres line with your IP address or network if you want to connect to your database remotely:
*filter
# Allow all loopback (lo0) traffic and drop all traffic to 127/8 that doesn't use lo0
-A INPUT -i lo -j ACCEPT
-A INPUT ! -i lo -d 127.0.0.0/8 -j REJECT
# Accept all established inbound connections
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# Allow all outbound traffic - you can modify this to only allow certain traffic
-A OUTPUT -j ACCEPT
# Allow HTTP and HTTPS connections from anywhere (the normal ports for websites and SSL).
-A INPUT -p tcp --dport 80 -j ACCEPT
-A INPUT -p tcp --dport 443 -j ACCEPT
# Allow ports for testing
-A INPUT -p tcp --dport 8080:8090 -j ACCEPT
# Allow ports for MOSH (mobile shell)
-A INPUT -p udp --dport 60000:61000 -j ACCEPT
# Allow SSH connections
# The -dport number should be the same port number you set in sshd_config
-A INPUT -p tcp -m state --state NEW --dport 22 -j ACCEPT
# Allow postgres connections, USE YOUR IP YOULL BE CONNECTING FROM HERE
-A INPUT -p tcp --dport 5432 -s YOURIPADDRESS/16 -j ACCEPT
# Allow ping
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT
# Log iptables denied calls
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7
# Reject all other inbound - default deny unless explicitly allowed policy
-A INPUT -j REJECT
-A FORWARD -j REJECT
COMMIT
sudo iptables-restore < /etc/iptables.firewall.rules
sudo iptables -L sudo vim /etc/network/if-pre-up.d/firewall
Now put the following text into the file;
#!/bin/sh /sbin/iptables-restore < /etc/iptables.firewall.rules
sudo chmod +x /etc/network/if-pre-up.d/firewall
Here are a couple useful lines that you don’t need to run now, but remembering case you need to check the firewall log:
iptables logging config
sudo vim /etc/rsyslog.conf
how to see iptables log
sudo tail -5000 /var/log/kern.log | grep DPT=80
Step 7 – Set up your database (PostgreSQL)
More info here, here, and here
First, we will use the psql tool to make a database user
sudo -u postgres psql template1
ALTER USER postgres with encrypted password 'YOURPASSWORDMD5HASH'; #You will need to md5 your password and put the hash here. You can find a website that will give you an md5 pretty easily
q
And add a working user and a database for your project
psql -U YOURDBUSER template1 CREATE DATABASE YOURPROJ;
q
Now, we need to edit the PostgreSQL configuration file to reflect the new user.
sudo vim /etc/postgresql/9.1/main/pg_hba.conf
local all postgres md5
sudo createuser -U postgres -d -e -E -l -P -r -s YOURPROJ
And now we edit the same configuration file to allow remote connections. Please be aware that this configuration allows anyone to connect to the database, which is not okay for production environments. If you want to restrict access, you’ll need to put in the IP address of your network that you’ll be working from.
sudo vim /etc/postgresql/9.1/main/pg_hba.conf
local all all md5 host all all 0.0.0.0/0 md5
Now edit the other PostgreSQL configuration file to tell it to listen for connections
sudo vim /etc/postgresql/9.1/main/postgresql.conf
listen_addresses = '*'
sudo /etc/init.d/postgresql restart
Step 8 – Source control with git and github
Go ahead and set up a Github account which will be our origin (main) repository. Now, tell git to put your project under source control, but we want to ignore our two virtualenvs and also compiled Python files (.pyc)
cd /home/YOURUSER/YOURPROJ
git init
sudo vim /home/YOURUSER/YOURPROJ/.gitignore
Enter this into the file:
YOURPROJ-env
fabric-env
*.pyc
git add .
git commit -a -m 'Initial commit of this project'
Now, we tell it that we want to use our Github account as our origin repository
git remote add origin https://github.com/YOURGITACCOUNT/YOURGITPROJ.git
Now that it knows about this origin, we are going to ‘push’ the commit that we just made locally above to Github
git push -u origin master
Step 9 – Create our pyramid project package
sudo /home/YOURUSER/YOURPROJ-env/bin/pcreate -s alchemy YOURPACKAGE #Whatever you want to call your initial package inside your project
sudo /home/YOURUSER/YOURPROJ/YOURPROJ-env/bin/python3 setup.py develop
sudo /home/YOURUSER/YOURPROJ/YOURPROJ-env/bin/python3 setup.py test -q
Edit the development.ini file for this new package. Note, that I am not going to put the database connection string in source control, so we are inserting a dummy connection string right now
vim /home/YOURUSER/YOURPROJ/YOURPACKAGE/development.ini
Set port to 8080
sqlalchemy.url = postgresql://YOURDBUSER:xxxxxx@localhost:5432/YOURPROJ
and while we are in this file, if you want the development toolbar, you’ll need to enter the IP address that you’ll be working from so the toolbar will show up:
debugtoolbar.hosts = xxxxxxxxxx #put in your IP address
Since we are not putting the database connection string inside of source control, will need to create a new file that will hold it
vim database.ini #in same directory as development.ini
[postgres]
connstring = postgresql://YOURDBUSER:xxxxxxxxxxxx@localhost:5432/YOURPROJ #You will put your real connection string here
We will need to go back to our .gitignore file and tell git to keep this new file out of the repository. We are also going to put in a database2.ini entry for our test database
sudo vim /home/YOURUSER/YOURPROJ/.gitignore
YOURPROJ-env
fabric-env
*.pyc
database.ini
database2.ini
When our project is started, we will need to enter some code to make the system load our actual database connection string and overwrite the dummy value. This is going to happen in the __init__.py file one directory down from development.ini. Imports at the top:
from configparser import SafeConfigParser import os
Now right under the def main(global_config, **settings):
parser = SafeConfigParser()
iniloc = os.path.abspath(os.path.join(os.path.dirname( __file__ ), '..', 'database.ini'))
read_list = parser.read(iniloc)
connstring = parser.get('postgres', 'connstring')
settings['sqlalchemy.url'] = connstring
Step 10 – Deployment (work in progress)
We are going to use this tool ‘fabric’ to script our deployment process. More info here.
So we open up the fabric script file
sudo vim /home/YOURUSER/YOURPROJ/fabfile.py
and put our deployment script in there.
For now, I’m just going to post it as is, but I will come back and explain it later. For this to work, you have to finish the database backup section below with the latest directory modifications.
from fabric.api import local from fabric.context_managers import lcd def backup_project(backup_date): local('sudo cp -R /home/YOURUSER/YOURPROJ/ /home/YOURUSER/backups/backup' + backup_date) def new_branch(branch_name, db): with lcd('/home/YOURUSER/dev'): #local('mkdir ' + branch_name) local('git clone https://github.com/YOURGITACCOUNT/YOURGITPROJ ' + branch_name) with lcd('/home/YOURUSER/dev/' + branch_name): local('sudo cp /home/YOURUSER/YOURPROJ/YOURPACKAGE/database' + db + '.ini ./YOURPACKAGE/database.ini') def refresh_test_db(db_name): if db_name != 'YOURPROJ': with settings(user="YOURUSER"): if local('psql -l -U postgres | grep ' + db_name + ' | wc -l') == 1: local('dropdb ' + db_name + ' -U postgres') local('createdb -T template0 ' + db_name + ' -U postgres') local('gunzip -c /home/YOURUSER/backups/database/latest/YOURPROJ.sql.gz > /home/YOURUSER/backups/database/latest/YOURPROJ.sql') local('psql -U postgres -d ' + db_name + ' -f /home/YOURUSER/backups/database/latest/YOURPROJ.sql') local('rm /home/YOURUSER/backups/database/latest/YOURPROJ.sql') def prepare_deployment(branch_name): local('/home/YOURUSER/YOURPROJ/YOURPROJ-env/bin/python3 /var/www/YOURPROJ/YOURPACKAGE/setup.py test -q') local('git add -p && git commit') local('git checkout master && git merge ' + branch_name) def deploy(branch_name): with lcd('/home/YOURUSER/YOURPROJ'): local('git pull /home/YOURUSER/dev/' + branch_name) #local('sudo service apache2 restart')
You will run these scripts like this:
/home/YOURUSER/YOURPROJ/fabric-env/bin/fab -f /home/YOURUSER/YOURPROJ/fabfile.py backup_project:date
/home/YOURUSER/YOURPROJ/fabric-env/bin/fab -f /home/YOURUSER/YOURPROJ/fabfile.py new_branch:test,2
/home/YOURUSER/YOURPROJ/fabric-env/bin/fab -f /home/YOURUSER/YOURPROJ/fabfile.py refresh_test_db:YOURTESTDBNAME
/home/YOURUSER/YOURPROJ/fabric-env/bin/fab -f /home/YOURUSER/YOURPROJ/fabfile.py prepare_deployment:test
/home/YOURUSER/YOURPROJ/fabric-env/bin/fab -f /home/YOURUSER/YOURPROJ/fabfile.py deploy:test
Step 11 – Database backups
Set up database backups with pg_backup.sh. More info here Edit pg_backup.config #Slight modifications to the script to allow for a latest directory. Keep in mind that you’ve got a make the modifications in pg_backup for this to include a latest directory.
mkdir /home/YOURUSER/backups/database/latest
Use pgpass to allow automatic script execution. More info here . make sure you are logged in as YOURUSER.
cd ~
touch .pgpass
vim .pgpass
*:*:*:YOURDBUSER:YOURDBPASS
chmod 0600 ~/.pgpass
bash /home/YOURUSER/YOURPROJ/YOURPACKAGE/YOURPACKAGE/scripts/pg_backup.sh
Run regularly with Cron
crontab -e
0 3 * * * /home/YOURUSER/YOURPROJ/YOURPACKAGE/YOURPACKAGE/scripts/pg_backup.sh > /dev/null 2>&1
check status
/etc/init.d/cron status
check the log
sudo grep CRON /var/log/syslog