Last week I met Stephan Diehl, Michael Hierweck, Veit Schiele, and Jens Vagelpohl\u00a0in Berlin for a sprint. Our chosen topic was “Python web application\u00a0deployment”. In this post I’d like to recap our discussions, gocept’s perspective on those, and the deployment tool “batou” that we have been incubating in the last months.<\/p>\n
<\/p>\n
All of us have experience both in writing web\u00a0applications with Python (Zope and Django specifically) and also have been\u00a0involved with system administration on varying complexity:<\/p>\n
Platforms and applications<\/strong><\/p>\n I expected a diverse, even controversial discussion about the challenges and Here is a list (of my choice) of ideas we discussed:<\/p>\n Ideally both IT and developers could make perfect use of their capabilities\u00a0which are often times overlapping and for which our existing stereotypical memes do not provide a good answer:\u00a0“Never let a developer touch a production box.”, “Sysadmins are those people that fix your\u00a0computer”.<\/p>\n I wish we were beyond that already.<\/li>\n I can haz DevOps manifesto, plz?<\/strong><\/p>\n Earlier this year I had a little epiphany and found re-reading the\u00a0“Principles behind the Agile Manifesto” to be extremely worthwhile – even if\u00a0you have been doing “agile development” for a while now. I was\u00a0surprised because I found them to be quite counter-intuitive to some of the\u00a0actions that I tend to do if things get rough. Maybe more on that later.<\/p>\n During the sprint I had a short moment when I wished there would also be a\u00a0“Devops Manifesto”. However, I think the principles mentioned above are\u00a0completely sufficient and reflect my Sysadmin\/DevOps experience quite well.<\/p>\n So, to everyone wishing to get their hands on something concisely written, that can\u00a0guide them when being responsible to (self-)organize: go, read the “Principles\u00a0behind the Agile Manifesto” and ponder them!<\/p>\n The gocept.net perspective<\/strong><\/p>\n We at gocept are running a platform (“gocept.net”) which tries to solve the\u00a0infrastructure-oriented problems and provide developers with a comfortable\u00a0environment. We avoid introducing funky new dingbats and instead provide our users with a well-managed operating system (Linux) and production-ready configurations of complex and\/or often-used components\u00a0(like PostgreSQL, Apache, haproxy, nginx, …) that can still be adjusted individually. This platform idea is our take on some of the thoughts that we also discussed at the sprint.<\/p>\n We sell the platform by accounting for the resources (CPU, RAM, disk, network)\u00a0that our customers require but as a customer you can always select and use any of the components available and immediately benefit from updates.<\/p>\n The platform has a reasonable user base considered that we are a company that\u00a0is not quite used to doing “products” yet. We are also used to growing organically\u00a0which in turn fits our agile approach.<\/p>\n However, although our platform provides a nice deployment target, we were\u00a0lacking an answer for developers to how they should drive their deployments.<\/p>\n Fortunately – we are moving towards an answer right now.<\/p>\n batou<\/strong><\/p>\n Meet: batou. (Yes, it’s the guy with the big guns from “Ghost in the Shell”\u00a0and also a friendly pun on Puppet’s “puppetmaster”.)<\/p>\n batou is a multi-(host|component|environment) deployment utility that is platform-independent and model-driven. We are aiming at something in the style of Puppet but from an application deployment perspective and we want it to be useful to developers and ops not \u00a0using our gocept.net platform.<\/p>\n We’re still early in the process of making it useful for people outside of gocept, so I would like to start with our context and history for developing it.<\/p>\n In the last 1.5 years we have been using Fabric to deploy our most complex\u00a0hosting project. It consists of about 30 VMs, using many different components:\u00a0haproxy, varnish, solr, Zope, ZEO, Postgresql, Tomcat, OpenLDAP, and others.<\/p>\n We were happy to have fabric around to get this deployment automated for\u00a0multiple environments but we have also been looking to get the sweet\u00a0configuration language that puppet provides into a tool like this.<\/p>\n As we started work on another hosting project earlier this year, we took our\u00a0experience from fabric, applied some heavy refactorings and ended up with a\u00a0more usable version that still leverages fabric internally.<\/p>\n The internal refactoring felt quite right but hasn’t been the big leap that we\u00a0were looking for. Luckily, during PyConUS, I spend some days trying to get my\u00a0head around a good and simple API do describe deployments and finally got a\u00a0break-through.<\/p>\n At the sprint I got the chance to introduce the current status of my still\u00a0experimental branch and we worked on some of the issues that I pointed out are\u00a0still hairy. We made some good progress on those parts and I’m quite happy\u00a0with where we’re going.<\/p>\n OK, so all this is quite “pie in the sky” for anyone not at gocept or not at\u00a0the sprint. I really need to write some documentation about batou to get this\u00a0usable for others.<\/p>\n However, until then, have a look at our sprint’s Bitbucket repository in which Take a special look at the files in “components” and “environments” as those\u00a0contain the interesting pieces.<\/p>\n That’s it for now and if batou should make any sense to you, feel free to\u00a0contact me for further discussions.<\/p>\n","protected":false},"excerpt":{"rendered":" Last week I met Stephan Diehl, Michael Hierweck, Veit Schiele, and Jens Vagelpohl\u00a0in Berlin for a sprint. Our chosen topic was “Python web application\u00a0deployment”. In this post I’d like to recap our discussions, gocept’s perspective on those, and the deployment tool “batou” that we have been incubating in the last months.<\/p>\n","protected":false},"author":12391367,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_coblocks_attr":"","_coblocks_dimensions":"","_coblocks_responsive_height":"","_coblocks_accordion_ie_support":"","advanced_seo_description":"","jetpack_seo_html_title":"","jetpack_seo_noindex":false,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_newsletter_tier_id":0,"footnotes":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false}}},"categories":[10221],"tags":[4121751,8915,832],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_likes_enabled":true,"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/pFP3y-2T","jetpack-related-posts":[{"id":1340,"url":"https:\/\/blog.gocept.com\/2013\/07\/31\/reproducable-automated-deployments-on-raspberrypi-with-batou\/","url_meta":{"origin":179,"position":0},"title":"Reproducable automated deployments on RaspberryPi with batou","author":"Daniel Havlik","date":"July 31, 2013","format":false,"excerpt":"For continuous integration during development, we use Jenkins to automatically run tests for all projects we maintain. Some time ago we wanted to increase visibility of the results, so we set up a Raspberry Pi driving a few meters of LPD8806-based LED strip on which we can address single LEDs\u2026","rel":"","context":"In "en"","block_context":{"text":"en","link":"https:\/\/blog.gocept.com\/category\/en\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1433,"url":"https:\/\/blog.gocept.com\/2014\/08\/04\/september-18th-20th-devops-sprint\/","url_meta":{"origin":179,"position":1},"title":"September, 18th\u201320th: DevOps Sprint","author":"Daniel Havlik","date":"August 4, 2014","format":false,"excerpt":"Since we have a strong history in web development, but also were involved in operating web applications we developed, the DevOps movement hit our nerves.Under the brand name \"Flying Circus\" we are establishing a platform respecting the DevOps principles.A large portion of our day-to-day work is dedicated to DevOps related\u2026","rel":"","context":"In "en"","block_context":{"text":"en","link":"https:\/\/blog.gocept.com\/category\/en\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3225,"url":"https:\/\/blog.gocept.com\/2018\/05\/24\/zope-is-welcome-in-the-python-3-wonderland\/","url_meta":{"origin":179,"position":2},"title":"Zope is welcome in the Python 3 wonderland!","author":"Michael Howitz","date":"May 24, 2018","format":false,"excerpt":"Earl Zope already got the beta permission to stay in the Python 3 wonderland some months ago. His current objective is to help old friends to come to the Python 3 wonderland and to make new friends. He has to build trust in his will and ability to stay in\u2026","rel":"","context":"In "en"","block_context":{"text":"en","link":"https:\/\/blog.gocept.com\/category\/en\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/blog.gocept.com\/wp-content\/uploads\/2018\/05\/zope-4-welcome-sprint.jpg?fit=1200%2C637&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.gocept.com\/wp-content\/uploads\/2018\/05\/zope-4-welcome-sprint.jpg?fit=1200%2C637&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/blog.gocept.com\/wp-content\/uploads\/2018\/05\/zope-4-welcome-sprint.jpg?fit=1200%2C637&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/blog.gocept.com\/wp-content\/uploads\/2018\/05\/zope-4-welcome-sprint.jpg?fit=1200%2C637&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/blog.gocept.com\/wp-content\/uploads\/2018\/05\/zope-4-welcome-sprint.jpg?fit=1200%2C637&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":1558,"url":"https:\/\/blog.gocept.com\/2016\/09\/05\/zope-in-the-python-3-wonderland\/","url_meta":{"origin":179,"position":3},"title":"Zope in the Python 3 wonderland","author":"Michael Howitz","date":"September 5, 2016","format":false,"excerpt":"A little tale Once upon the time there was the\u00a0big mighty Zope II. It was one of the leaders in the Python land. It had mighty features like TTW (trough the web) development\u00a0and its own object oriented database. Many\u00a0people liked Zope II and trusted it to be the basis for\u2026","rel":"","context":"In "en"","block_context":{"text":"en","link":"https:\/\/blog.gocept.com\/category\/en\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1752,"url":"https:\/\/blog.gocept.com\/2016\/10\/04\/zope-resurrection-part-2-defibrillation\/","url_meta":{"origin":179,"position":4},"title":"Zope Resurrection Part 2 \u2013 Defibrillation","author":"Michael Howitz","date":"October 4, 2016","format":false,"excerpt":"After\u00a0reanimation\u00a0we started\u00a0defibrillation of Zope and \u2026 it kinda worked: On our\u00a0sprint we got\u00a0the following things done to help\u00a0Zope in the Python 3 wonderland: Release\u00a0zope.testbrowser 5.0\u00a0which is compatible with Python 3 and no longer uses\u00a0mechanize. PullRequest for\u00a0ExtensionClass\u00a0with the Python 3 port of the C extension. five.globalrequest\u00a0is now compatible with Python 3.\u2026","rel":"","context":"In "en"","block_context":{"text":"en","link":"https:\/\/blog.gocept.com\/category\/en\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/blog.gocept.com\/wp-content\/uploads\/2016\/10\/zopesprint.jpg?fit=1200%2C709&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.gocept.com\/wp-content\/uploads\/2016\/10\/zopesprint.jpg?fit=1200%2C709&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/blog.gocept.com\/wp-content\/uploads\/2016\/10\/zopesprint.jpg?fit=1200%2C709&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/blog.gocept.com\/wp-content\/uploads\/2016\/10\/zopesprint.jpg?fit=1200%2C709&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/blog.gocept.com\/wp-content\/uploads\/2016\/10\/zopesprint.jpg?fit=1200%2C709&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":3212,"url":"https:\/\/blog.gocept.com\/2018\/04\/03\/a-heartily-welcome-for-zope-in-the-python-3-wonderland\/","url_meta":{"origin":179,"position":5},"title":"A heartily welcome for Zope in the Python 3 wonderland","author":"Michael Howitz","date":"April 3, 2018","format":false,"excerpt":"Once upon the time there was Earl Zope II. A wise guy was telling him that his world will come to an end. He found out that this was true that he had only some years to prepare to immigrate to the Python 3 wonderland. His preparation was successful: He\u2026","rel":"","context":"In "en"","block_context":{"text":"en","link":"https:\/\/blog.gocept.com\/category\/en\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/blog.gocept.com\/wp-content\/uploads\/2018\/03\/cathal-mac-an-bheatha-223618-unsplash-e1522239717994.jpg?fit=1017%2C1200&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/blog.gocept.com\/wp-content\/uploads\/2018\/03\/cathal-mac-an-bheatha-223618-unsplash-e1522239717994.jpg?fit=1017%2C1200&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/blog.gocept.com\/wp-content\/uploads\/2018\/03\/cathal-mac-an-bheatha-223618-unsplash-e1522239717994.jpg?fit=1017%2C1200&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/blog.gocept.com\/wp-content\/uploads\/2018\/03\/cathal-mac-an-bheatha-223618-unsplash-e1522239717994.jpg?fit=1017%2C1200&ssl=1&resize=700%2C400 2x"},"classes":[]}],"_links":{"self":[{"href":"https:\/\/blog.gocept.com\/wp-json\/wp\/v2\/posts\/179"}],"collection":[{"href":"https:\/\/blog.gocept.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.gocept.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.gocept.com\/wp-json\/wp\/v2\/users\/12391367"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.gocept.com\/wp-json\/wp\/v2\/comments?post=179"}],"version-history":[{"count":9,"href":"https:\/\/blog.gocept.com\/wp-json\/wp\/v2\/posts\/179\/revisions"}],"predecessor-version":[{"id":185,"href":"https:\/\/blog.gocept.com\/wp-json\/wp\/v2\/posts\/179\/revisions\/185"}],"wp:attachment":[{"href":"https:\/\/blog.gocept.com\/wp-json\/wp\/v2\/media?parent=179"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.gocept.com\/wp-json\/wp\/v2\/categories?post=179"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.gocept.com\/wp-json\/wp\/v2\/tags?post=179"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\npossible solutions that everybody sees – but surprisingly, we pretty much agreed on those issues right from the beginning.<\/p>\n\n
\nwe tried to write a deployment for Django’s “Hello world”<\/a> application.\u00a0\u00a0Also, as batou is open source, of course, you can dive right into the code<\/a>.<\/p>\n