Last week I met Stephan Diehl, Michael Hierweck, Veit Schiele, and Jens Vagelpohl in Berlin for a sprint. Our chosen topic was “Python web application deployment”. In this post I’d like to recap our discussions, gocept’s perspective on those, and the deployment tool “batou” that we have been incubating in the last months.
All of us have experience both in writing web applications with Python (Zope and Django specifically) and also have been involved with system administration on varying complexity:
- Jens formerly was a Sysadmin at Zope Corporation
- Michael works with the Hostsharing e.G.
- Veit has been consulting with various companies that operate their own systems
- Stephan is working on the website of Ableton
- I have been working on gocept.net and our various hosting projects at gocept for some years
Platforms and applications
I expected a diverse, even controversial discussion about the challenges and
possible solutions that everybody sees – but surprisingly, we pretty much agreed on those issues right from the beginning.
Here is a list (of my choice) of ideas we discussed:
- The traditional separation of admins (IT) and developers is broken.Everyone seems to strive for a model where one group will provide an infrastructure or platform (as in IaaS or PaaS) and another group can use that platform as the interface to run applications on top.The interesting challenge is to find a good definition of that interface: should it be more on the infrastructure-level (like AWS) or a more polished platform (like Heroku or GAE)?In any case: groups that tend to work with similar tools, should still be separated by their responsibilities: just because there it is a piece of software running on a server not mean your desk-and-printer-and-network IT is the right group to be responsible for it.
Ideally both IT and developers could make perfect use of their capabilities which are often times overlapping and for which our existing stereotypical memes do not provide a good answer: “Never let a developer touch a production box.”, “Sysadmins are those people that fix your computer”.
I wish we were beyond that already.
- It’s 2012 – if your response to the idea of running many VMs is “it is too expensive” then you are clearly doing something wrong.
- “Cloud” and cloud solutions should not be the reason to neglect or reinvent good practices that have been established previously and that have a community.Most of us run Unix-ish environments and the practices that are valid there are becoming even more valid and approachable with infrastructure-automation ala Puppet or Chef.
- A solution to a problem that starts with “Buy our product” usually neglects the fact that neither software development nor operations have silver bullets.However, and this caters to the platform idea, we should make existing known problems easier to solve and/or understand. This does involve documentation of technology and processes as well as better tooling as well as making processes simpler.
- If you are doing anything Sysadmin or Ops, you really should buy and read “The practice of system and network administration”.
I can haz DevOps manifesto, plz?
Earlier this year I had a little epiphany and found re-reading the “Principles behind the Agile Manifesto” to be extremely worthwhile – even if you have been doing “agile development” for a while now. I was surprised because I found them to be quite counter-intuitive to some of the actions that I tend to do if things get rough. Maybe more on that later.
During the sprint I had a short moment when I wished there would also be a “Devops Manifesto”. However, I think the principles mentioned above are completely sufficient and reflect my Sysadmin/DevOps experience quite well.
So, to everyone wishing to get their hands on something concisely written, that can guide them when being responsible to (self-)organize: go, read the “Principles behind the Agile Manifesto” and ponder them!
The gocept.net perspective
We at gocept are running a platform (“gocept.net”) which tries to solve the infrastructure-oriented problems and provide developers with a comfortable environment. We avoid introducing funky new dingbats and instead provide our users with a well-managed operating system (Linux) and production-ready configurations of complex and/or often-used components (like PostgreSQL, Apache, haproxy, nginx, …) that can still be adjusted individually. This platform idea is our take on some of the thoughts that we also discussed at the sprint.
We sell the platform by accounting for the resources (CPU, RAM, disk, network) that our customers require but as a customer you can always select and use any of the components available and immediately benefit from updates.
The platform has a reasonable user base considered that we are a company that is not quite used to doing “products” yet. We are also used to growing organically which in turn fits our agile approach.
However, although our platform provides a nice deployment target, we were lacking an answer for developers to how they should drive their deployments.
Fortunately – we are moving towards an answer right now.
batou
Meet: batou. (Yes, it’s the guy with the big guns from “Ghost in the Shell” and also a friendly pun on Puppet’s “puppetmaster”.)
batou is a multi-(host|component|environment) deployment utility that is platform-independent and model-driven. We are aiming at something in the style of Puppet but from an application deployment perspective and we want it to be useful to developers and ops not using our gocept.net platform.
We’re still early in the process of making it useful for people outside of gocept, so I would like to start with our context and history for developing it.
In the last 1.5 years we have been using Fabric to deploy our most complex hosting project. It consists of about 30 VMs, using many different components: haproxy, varnish, solr, Zope, ZEO, Postgresql, Tomcat, OpenLDAP, and others.
We were happy to have fabric around to get this deployment automated for multiple environments but we have also been looking to get the sweet configuration language that puppet provides into a tool like this.
As we started work on another hosting project earlier this year, we took our experience from fabric, applied some heavy refactorings and ended up with a more usable version that still leverages fabric internally.
The internal refactoring felt quite right but hasn’t been the big leap that we were looking for. Luckily, during PyConUS, I spend some days trying to get my head around a good and simple API do describe deployments and finally got a break-through.
At the sprint I got the chance to introduce the current status of my still experimental branch and we worked on some of the issues that I pointed out are still hairy. We made some good progress on those parts and I’m quite happy with where we’re going.
OK, so all this is quite “pie in the sky” for anyone not at gocept or not at the sprint. I really need to write some documentation about batou to get this usable for others.
However, until then, have a look at our sprint’s Bitbucket repository in which
we tried to write a deployment for Django’s “Hello world” application. Also, as batou is open source, of course, you can dive right into the code.
Take a special look at the files in “components” and “environments” as those contain the interesting pieces.
That’s it for now and if batou should make any sense to you, feel free to contact me for further discussions.