Growing our bootstrap script

Our company currently is using a custom bootstrap script to install our application which is based on virtualenv. Our 'bootstrapper' does a few things:

1. Sets up virtualenv
2. Runs configuration scripts
3. Other stuff

While our script works and has served us well, when cleaning it up and trying to expand it's functionality it began to show it's cruftiness, the kind of whiff you get that maybe the script itself needs the help of some third party libraries. Two of the python libraries that I have seen that can help us are:

• virtualenv-tools to help us with cloning our virtualenvs
• unipath or py-path to clean up the parts of the code relying on os.path and gang (ugh really hate those)

It seemed simple enough initially, just add these libraries or third party applications and ensure that we install them on our live servers as well. So, there I was a confident idiot, accessing our production servers installing the required pieces, checking in the modified code ... going home for the day thinking that all was well. Well, all was not good and dandy. The next day my QA department looked at me with this crossed look because all of her test servers broke. Well of course they broke, they too used the same bootstrap script and they too needed to be installed with the new required pieces. "There must be a better way of doing this" I silently thought to myself, then I remembered testing out this application building system from the Zope foundation called "buildout". I wondered if it would be useful for me. I read through the docs and it seems it was just what I needed!

After numerous false starts (The acquaintance process took longer than I anticipated as grasping it's core concepts was not as easy as I thought). So now here is how I do our bootstrapping using buildout, I create a buildout.cfg with barebone contents:

[buildout]
parts = cot

[cot]
recipe = zc.recipe.egg
interpreter = python
eggs =
    virtualenv-tools
    virtualenv
    configobj

Then I run

python bootstrap.py -d

then to finally pull in my bootstrap's library requirements in the form of local eggs.

bin/bootstrap

In this way it's self contained and I need not touch any of the system's python's libraries. Nice! Finally I can run my bootstrapper with:

bin/python mysuperbootstrapper.py

Then after the bootstrapper runs it's course, I can activate my virtualenv and hand the reigns there. No need for me to manually go in and ensure that all of my bootstrap's system requirements of python libraries are met. Done!

There are few places where this workflow can improve, one that comes right off the bat is why not just make the whole thing a buildout process. Well, I am looking into that as there are more than a few places where our setup.py files has to be rethought and better-ed. I am looking at that and hoping to reshape our whole deployment plus bootstrapping process hopefully based on buildout. I am also thinking of reshaping our testing process around buildout to get rid of retarded stuff like installing nosetests on production servers.

I will post my findings on our progress in the migration process ... as for now I am complete.