Last week I met Stephan Diehl, Michael Hierweck, Veit Schiele, and Jens Vagelpohl\u00a0in Berlin for a sprint. Our chosen topic was “Python web application\u00a0deployment”. In this post I’d like to recap our discussions, gocept’s perspective on those, and the deployment tool “batou” that we have been incubating in the last months.<\/p>\n
<\/p>\n
All of us have experience both in writing web\u00a0applications with Python (Zope and Django specifically) and also have been\u00a0involved with system administration on varying complexity:<\/p>\n
Platforms and applications<\/strong><\/p>\n
I expected a diverse, even controversial discussion about the challenges and
\npossible solutions that everybody sees – but surprisingly, we pretty much agreed on those issues right from the beginning.<\/p>\n
Here is a list (of my choice) of ideas we discussed:<\/p>\n
Ideally both IT and developers could make perfect use of their capabilities\u00a0which are often times overlapping and for which our existing stereotypical memes do not provide a good answer:\u00a0“Never let a developer touch a production box.”, “Sysadmins are those people that fix your\u00a0computer”.<\/p>\n
I wish we were beyond that already.<\/li>\n
I can haz DevOps manifesto, plz?<\/strong><\/p>\n
Earlier this year I had a little epiphany and found re-reading the\u00a0“Principles behind the Agile Manifesto” to be extremely worthwhile – even if\u00a0you have been doing “agile development” for a while now. I was\u00a0surprised because I found them to be quite counter-intuitive to some of the\u00a0actions that I tend to do if things get rough. Maybe more on that later.<\/p>\n
During the sprint I had a short moment when I wished there would also be a\u00a0“Devops Manifesto”. However, I think the principles mentioned above are\u00a0completely sufficient and reflect my Sysadmin\/DevOps experience quite well.<\/p>\n
So, to everyone wishing to get their hands on something concisely written, that can\u00a0guide them when being responsible to (self-)organize: go, read the “Principles\u00a0behind the Agile Manifesto” and ponder them!<\/p>\n
The gocept.net perspective<\/strong><\/p>\n
We at gocept are running a platform (“gocept.net”) which tries to solve the\u00a0infrastructure-oriented problems and provide developers with a comfortable\u00a0environment. We avoid introducing funky new dingbats and instead provide our users with a well-managed operating system (Linux) and production-ready configurations of complex and\/or often-used components\u00a0(like PostgreSQL, Apache, haproxy, nginx, …) that can still be adjusted individually. This platform idea is our take on some of the thoughts that we also discussed at the sprint.<\/p>\n
We sell the platform by accounting for the resources (CPU, RAM, disk, network)\u00a0that our customers require but as a customer you can always select and use any of the components available and immediately benefit from updates.<\/p>\n
The platform has a reasonable user base considered that we are a company that\u00a0is not quite used to doing “products” yet. We are also used to growing organically\u00a0which in turn fits our agile approach.<\/p>\n
However, although our platform provides a nice deployment target, we were\u00a0lacking an answer for developers to how they should drive their deployments.<\/p>\n
Fortunately – we are moving towards an answer right now.<\/p>\n
batou<\/strong><\/p>\n
Meet: batou. (Yes, it’s the guy with the big guns from “Ghost in the Shell”\u00a0and also a friendly pun on Puppet’s “puppetmaster”.)<\/p>\n
batou is a multi-(host|component|environment) deployment utility that is platform-independent and model-driven. We are aiming at something in the style of Puppet but from an application deployment perspective and we want it to be useful to developers and ops not \u00a0using our gocept.net platform.<\/p>\n
We’re still early in the process of making it useful for people outside of gocept, so I would like to start with our context and history for developing it.<\/p>\n
In the last 1.5 years we have been using Fabric to deploy our most complex\u00a0hosting project. It consists of about 30 VMs, using many different components:\u00a0haproxy, varnish, solr, Zope, ZEO, Postgresql, Tomcat, OpenLDAP, and others.<\/p>\n
We were happy to have fabric around to get this deployment automated for\u00a0multiple environments but we have also been looking to get the sweet\u00a0configuration language that puppet provides into a tool like this.<\/p>\n
As we started work on another hosting project earlier this year, we took our\u00a0experience from fabric, applied some heavy refactorings and ended up with a\u00a0more usable version that still leverages fabric internally.<\/p>\n
The internal refactoring felt quite right but hasn’t been the big leap that we\u00a0were looking for. Luckily, during PyConUS, I spend some days trying to get my\u00a0head around a good and simple API do describe deployments and finally got a\u00a0break-through.<\/p>\n
At the sprint I got the chance to introduce the current status of my still\u00a0experimental branch and we worked on some of the issues that I pointed out are\u00a0still hairy. We made some good progress on those parts and I’m quite happy\u00a0with where we’re going.<\/p>\n
OK, so all this is quite “pie in the sky” for anyone not at gocept or not at\u00a0the sprint. I really need to write some documentation about batou to get this\u00a0usable for others.<\/p>\n
However, until then, have a look at our sprint’s Bitbucket repository in which
\nwe tried to write a deployment for Django’s “Hello world”<\/a> application.\u00a0\u00a0Also, as batou is open source, of course, you can dive right into the code<\/a>.<\/p>\n