Richard Bucker

My New Python Project Setup

Posted at — Dec 30, 2011

[update 2012-01-18] postgres has been updated to 9.1.2; the latest version as of today.[update 2012-01-17] feel free to ignore my comments about Lua. While Lua might sit in an interesting place between Python and Java in an embedded/scripting place. The fact that lunatic-python does not compile and lupa depends on LuaJIT2 which is compatible with Lua 5.1 and the current Lua version 5.2 was recently released… and the comment from the LuaJIT team about adoption was a little snarky. I gotta think about something else.[update 2011-12-29] I forgot to add twitter’s bootstrap CSS/JS. I’ll cover that in a future post when I also discuss modern-package-templateIt’s pretty simple to set things up. There are some prerequisites and some basic install packages that need root access but the intent is to get the config in userspace as soon as possible. This article covers VM slices at Rackspace using Ubuntu 11.10.First: Install and update: allocate the OS select the OS and wait for it to complete.  You’re going to receive an email with the root password login, change the root password create an “admin” privileged user (usually my name or “builder”) add this user to the sudo change it’s password edit /etc/ssh/sshd_config and disable root login update the package definitions (apt-get update) upgrade the packages (apt-get upgrade) rebootSecond: Install the required roo packages postfix - when prompted select the default values apt-get -y install postfix apt-get -y install python-setuptools daemontools daemontools-run python-dev mailutils mutt build-essential uuid-dev python-nose vim htop sysstat dstat ifstat screen locate apache2-utils unzip siege python-virtualenv bwm-ng libcairo2-dev libglib2.0-dev libpango1.0-dev libxml2-dev fail2ban openssl libssl-doc openvpn libssl-dev libgcrypt11-dev lighttpd lighttpd-dev libevent-dev libcurl4-openssl-dev  libreadline6-dev beanstalkd tree apt-get install postgresql-9.1 postgresql-client-9.1 postgresql-doc-9.1 postgresql-plperl-9.1 postgresql-plpython-9.1 postgresql-server-dev-9.1 easy_install pip easy_install mercurial easy_install pycurl pip install virtualenvwrapperThat’s it, essentially, for the second layer, however, here’s an explanation of the modules from a macro perspective: python-setuptools - make the installer, easy_install, available daemontools daemontools-run - there are so many ways to implement a ‘daemon’ this tools make it simple to make daemon deployment simple python-dev python-nose python-virtualenv - basic prereqs for python development. virtualenv is needed so that packages can be installed in userspace mailutils mutt - generate emails build-essential uuid-dev  - basic developer tools vim screen - editor and console tool htop sysstat dstat ifstat locate unzip bwm-ng - debug /monitoring tools libcairo2-dev libglib2.0-dev libpango1.0-dev libxml2-dev - libs used when rendering usage graphics fail2ban - detect login attempts and put the IP in time-out openssl libssl-doc openvpn libssl-dev libgcrypt11-dev libcurl4-openssl-dev - crypto lighttpd lighttpd-dev - web server that should be in front of the framework libevent-dev - kevent, kpoll libs apache2-utils siege - performance simulation tools beanstalkd - message queueAnd finally the third layer, the userspace framework layer. But before you start installing packages you need to create the virtual environment: cd ${HOME} mkdir -p src cd ${HOME}/src virtualenv currentenv . ./currentenv/bin/activateNow install the third layer. pip install tornado pip install flask pip install flask-rest easy_install pip pip install pycurl pip install simplejson pip install tornado pip install Fabric pip install PasteDeploy pip install PasteScript pip install modern-package-template pip install requests pip install gevent pip install pystache pip install nose pip install redis-py pip install pymongo pip install hoover pip install pyzmq pip install pyyaml pip install beanstalkc pip install django pip install django-redis-cache pip install clint pip install djangorestframework pip install pyparsing pip install flupI’m hoping that there is a practical use-case for embedding Lua in Python. There are an few interesting projects like lunatic-python and lupa. Normally I would not consider Lua for anything beyond “hello world”, however, the redis team is embedding Lua, it seems like a very lightweight codebase, it can be embedded in just about any language (do a google search). cd ${HOME} mkdir -p tmp cd tmp wget http://www.lua.org/ftp/lua-5.2.0.tar.gz tar zxvf lua-5.2.0.tar.gz cd lua-5.2.0 make linux sudo make installNOTE: the lunatic project does not compile under Lua 5.2. So this thread is postponed for now. pip install lunatic-pythonAlternatively I tried [lupa] but that requires LuaJIT 2.0 which is currently in beta (version 9) cd ${HOME} mkdir -p tmp cd tmp wget http://luajit.org/download/LuaJIT-2.0.0-beta9.tar.gz tar zxvf LuaJIT-2.0.0-beta9.tar.gz cd LuaJIT-2.0.0-beta9 make sudo make install sudo ldconfigThen install lupa. pip install lupaNOTE: hoover is a client library for loggly.com.  You’ll need an account if you want to use this service.In closing, I would like to include a few more libraries, however, the current version in apt-get is too old. I’d prefer installing them from scratch. They are necessary packages so for the time-being I’m just going to list them. They should be installed when installing the first layer and by the root user (or sudo) ZeroMQ - trivial to build and deploy if you follow the instructions cd /tmp wget http://download.zeromq.org/zeromq-2.1.11.tar.gz tar zxvf zeromq-2.1.11.tar.gz ./configure make make install ldconfig MongoDB - (can actually be installed in userspace) cd /tmp wget http://fastdl.mongodb.org/osx/mongodb-linux-x86_64-2.0.2.tgz mkdir -p ${HOME}/bin cd ${HOME}/bin tar zxvf mongodb-linux-x86_64-2.0.2.tgz sudo mkdir -p /data/db sudo chown id -u /data/db ./mongodb-linux-x86_64-2.0.2/bin/mongod … or … cd ${HOME}/bin find ./mongodb-linux-x86_64-2.0.2/bin/ -type f -exec ln -s {} ; ./mongod Redis - trivial to build and deploy if you follow the instructions cd /tmp wget http://redis.googlecode.com/files/redis-2.4.6.tar.gz tar zxvf redis-2.4.6.tar.gz cd redis-2.4.6 make make install SQLite - a simple SQL DB cd /tmp wget http://www.sqlite.org/sqlite-autoconf-3070900.tar.gz tar zxvf sqlite-autoconf-3070900.tar.gz cd sqlite-autoconf-3070900 ./configure make make install ldconfig ISO8583 - ISO8583 lib # will use virtualenv if it is configured cd /tmp http://pypi.python.org/packages/source/I/ISO8583%20Module/ISO8583%20Module-1.2.zip#md5=259c5eb63bb36e3376f8a430a5b4a092 unzip ISO8583\ Module-1.2.zip cd ISO8583\ Module-1.2 python ./setup.py installGood luck!PS: You should consider scripting this installation so that the deploy can be automated.  Specially via Fabris, chef, or puppet.