Python 2 and 3 compatible builds with zc.buildout

Creating a single-source build environment with zc.buildout that works for both Python 2 and 3 is a bit of a hassle. This blog post shows how to do it for a minimal demo project.

During the sprints at PyCon DE 2012, we tried to make the upcoming 1.0 release of the nagiosplugin library compatible with both Python 2.7 and Python 3.2. Going for a single code base (without preprocessing steps like 3to2) was no too hard. The only thing left was a single-source zc.buildout setup suited for both Python 2.7 and 3.2. It worked out at last, but currently it needs two buildout configurations. This is a little bit kludgy. I hope that things will improve in the near future so that a single-source build environment with zc.buildout will be possible.

In the following, I will demonstrate the steps with a simple demo project called MultiVersion. It contains nothing more than a single class that is supposed to run under both Python 2 and 3. There is also a unit test to verify that the code works. We use zope.testrunner to run the unit tests. The code’s functionality is irrelevant for the examples, so I left it out. You can download the full source if you are interested.

1. Use a recent enough virtualenv

Older versions of virtualenv are generally not suited since they ship with obsolete releases of distribute and pip. Check if the virtualenv included in your GNU/Linux distribution is too old. Anything below 1.8 reduces the chance of success, so better install a current virtualenv locally then. Likewise, our bootstrap.py must be recent enough to support both Python 2 and 3. The standard bootstrap.py from python-distribute.org does currently not work with Python 3.

Now we are ready to create a virtualenv in a fresh source checkout.

Python 3.2:

$ virtualenv -p python3.2 .
Running virtualenv with interpreter /usr/bin/python3.2
New python executable in ./bin/python3.2
Installing distribute.....done.
Installing pip.....done.

Python 2.7:

$ virtualenv -p python2.7 .
Running virtualenv with interpreter /usr/bin/python2.7
New python executable in ./bin/python2.7
Not overwriting existing python script ./bin/python (you must use ./bin/python2.7)
Installing setuptools.....done.
Installing pip.....done.

2. Running buildout with Python 3.2

I will discuss the steps for Python 3.2 first, since main development will concentrate on newer Python versions. After that, I will describe the necessary steps to make the build environment backward compatible.

To run zc.buildout, we need a buildout.cfg file. I prefer to pin package versions in all projects to ensure reliable builds. As of writing this blog post, there is just an alpha release of zc.buildout that supports Python 3.2. Unfortunately, this version of zc.buildout supports Python 3.2 only, so don’t try this with Python 3.3.

My basic buildout.cfg looks like this:

[buildout]
allow-picked-versions = false
develop = .
newest = false
package = multiversion
parts = multiversion test
versions = versions

[versions]
distribute = 0.6.28
z3c.recipe.scripts = 1.0.1
zc.buildout = 2.0.0a2
zc.recipe.egg = 2.0.0a2
zc.recipe.testrunner = 1.4.0
zope.exceptions = 4.0.1
zope.interface = 4.0.1
zope.testrunner = 4.0.4

[multiversion]
recipe = zc.recipe.egg
eggs = ${buildout:package}
interpreter = py

[test]
recipe = zc.recipe.testrunner
eggs = ${buildout:package}
defaults = ['--auto-color']

In my experience, it is best to pin distutils to exactly the same version that is included in virtualenv’s support files. While differing versions are possible, they may trigger hard to find bugs since it is not always clear which version is used is which step.

I use the Python interpreter from my virtualenv’s bin directory while creating the buildout executable. This saves me from using activate/deactivate scripts which are slightly cumbersome in my opinion.

$ bin/python3.2 bootstrap.py
Creating directory 'blog-python-2-3/parts'.
Creating directory 'blog-python-2-3/develop-eggs'.
Generated script 'blog-python-2-3/bin/buildout'.

$ bin/buildout
Develop: 'blog-python-2-3/.'
Installing multiversion.
Generated interpreter 'blog-python-2-3/bin/py'.
Installing test.
Generated script 'blog-python-2-3/bin/test'.

Now we have a working build for Python 3.2:

$ bin/test
Running zope.testrunner.layer.UnitTests tests:
  Set up zope.testrunner.layer.UnitTests in 0.000 seconds.
  Ran 1 tests with 0 failures and 0 errors in 0.002 seconds.
Tearing down left over layers:
  Tear down zope.testrunner.layer.UnitTests in 0.000 seconds.

3. Running buildout with Python 2.7

Unfortunately, the current zc.buildout alpha release does not work with anything except Python 3.2. Running bootstrap.py fails:

$ bin/python2.7 bootstrap.py
Getting distribution for 'zc.buildout==2.0.0a2'.
While:
  Bootstrapping.
  Getting distribution for 'zc.buildout==2.0.0a2'.
Error: Couldn't find a distribution for 'zc.buildout==2.0.0a2'.

There is no single zc.buildout distribution that fits both Python 2.7 and 3.2. To get around this, I need to create a special-case buildout.cfg that changes version pinnings for incompatible packages. Besides zc.buildout, zc.recipe.egg needs different versions for Python 2.7 and 3.2 as well.

I create buildout-2.x.cfg (slightly grumbling):

[buildout]
extends = buildout.cfg

[versions]
zc.buildout = 1.6.3
zc.recipe.egg = 1.3.2

This one does the job when used with both bootstrap and buildout:

$ bin/python2.7 bootstrap.py -c buildout-2.x.cfg
Generated script 'blog-python-2-3/bin/buildout'.

$ bin/buildout -c buildout-2.x.cfg
Develop: 'blog-python-2-3/.'
Installing multiversion.
Generated interpreter 'blog-python-2-3/bin/py'.
Installing test.
Generated script 'blog-python-2-3/bin/test'.

We now have a build environment that builds single-source code for both Python 2.7 and 3.2 using zc.buildout. Of course, this technique could be extended to support even more versions. But I hope that the incompatible packages will be updated in the near future so that the need for special-case buildout.cfg files will go away. What seems to be most missing: a release of zc.buildout that supports all major Python versions.

TL;DR

  • Use a current virtualenv version.
  • Use a compatible bootstrap.py.
  • Pin your package versions.
  • Versions for some packages (including zc.buildout) must be special-cased.

Acknowledgements

I would like to thank Andrei Chirila and Michael Howitz for a great sprint session.

Sprint report: Deploying Python web applications – platforms and applications

Last week I met Stephan Diehl, Michael Hierweck, Veit Schiele, and Jens Vagelpohl in Berlin for a sprint. Our chosen topic was “Python web application deployment”. In this post I’d like to recap our discussions, gocept’s perspective on those, and the deployment tool “batou” that we have been incubating in the last months.

Continue reading “Sprint report: Deploying Python web applications – platforms and applications”

Custom widgets in zope.formlib

zope.formlib has the ability to customize the used widget like this:

class KeywordsManagementForm(five.formlib.formbase.SubPageForm):
    form_fields = zope.formlib.form.Fields(IKeywords)
    form_fields['keywords'].custom_widget = KWSelectWidgetFactory

I do not like this approach for two reasons:

  • the widget has to be set manually every time the specific field is used
  • there is no easy way to get a display widget if the form or field is not editable for the user

Defining a new schema field and registering the widget for this field seems a bit heavy, so I came up with providing a marker interface on the field:

class IHaveSelectableKeywords(zope.interface.Interface):
    """Marker interface to get a special keywords widget."""

class IKeywords(zope.interface.Interface):
    keywords = zope.schema.List(
        title = _("Edit Keywords"),
        value_type = zope.schema.Choice(
            vocabulary=u"uc.keywords.Keywords"))
    zope.interface.alsoProvides(keywords, IHaveSelectableKeywords)

I registered the edit widget and display widget for the IHaveSelectableKeywords interface, so the custom widget does not have to be set in the form like this (edit widget):

<adapter
   for=".IHaveSelectableKeywords
        zope.publisher.interfaces.browser.IBrowserRequest"
 provides="zope.app.form.browser.interfaces.ISimpleInputWidget"
 factory=".KWSelectWidgetFactory"
 permission="zope.Public" />

Sprint fruits: gocept.exttest and gocept.package

The whole company spent three days in Kloster Drübeck sprinting on internal tools and topics.

We overhauled our workflow for generating invoices and identified steps that we could automate.

We polished and released gocept.exttest, which integrates for example JavaScript unittest to Python’s unittest framework. In a nutshell, it allows you to write JS tests like this:

require 'my_app.js'

describe 'MyApp', ->
  it 'has read Douglas Adams', ->
  expect(new MyApp().calculate_the_answer()).toEqual(42)

and include them into your Python test suite with a single command:

import gocept.exttest
def test_suite():
    return gocept.exttest.makeSuite(
        os.environ.get('jasmine-bin'),
        '--coffee',
        '--json',
        pkg_resources.resource_filename('your.package', 'tests'))

The third area of our efforts was documentation, we designed a Sphinx skeleton to make it easy to get started writing docs, and created a template for eggs that contains the necessary boilerplate and codifies our packaging and documentation conventions. While the concrete details are probably a bit specific to our tastes, some of the general mechanincs might be interesting to others, so we’ll release gocept.package once we’ve got the missing integration tests sorted out.

Shutting down an HTTPServer

For integration tests it can be helpful to have a fake HTTP server whose behaviour the tests can control. All necessary building blocks are even included in Python standard library. However, the BaseHTTPServer is surprisingly hard to shut down properly, so that it gives up the socket and everything.

While working on gocept.selenium, we came up with some code that does the trick (together with Jan-Wijbrand Kolman and Jan-Jaap Driessen).

class HTTPServer(BaseHTTPServer.HTTPServer):

    _continue = True

    def serve_until_shutdown(self):
        while self._continue:
            self.handle_request()

    def shutdown(self):
        self._continue = False
        # We fire a last request at the server in order to take it out of the
        # while loop in `self.serve_until_shutdown`.
        try:
            urllib2.urlopen(
                'http://%s:%s/' % (self.server_name, self.server_port))
        except urllib2.URLError:
            # If the server is already shut down, we receive a socket error,
            # which we ignore.
            pass
        self.server_close()

You might use this in a zope.testrunner layer like this:

class SilentRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):

    def log_message(self, format, *args):
        pass


class HTTPServerLayer(object):

    host = 'localhost'

    def setUp(self):
        self.server = None
        self.port = random.randint(30000, 40000)
        self.start_server()

    def start_server(self):
        self.server = HTTPServer((self.host, self.port), SilentRequestHandler)
        self.server_thread = threading.Thread(
            target=self.server.serve_until_shutdown)
        self.server_thread.daemon = True
        self.server_thread.start()
        # Kludge: Wait a little as it sometimes takes a while to get the server
        # started.
        time.sleep(0.25)

    def stop_server(self):
        if self.server is None:
            return
        self.server.shutdown()
        self.server_thread.join()

    def tearDown(self):
        self.stop_server()

 

 

Get a zc.sourcefactory to implement an interface

zc.sourcefactory is very handy to easily create a source (zope.schema.interfaces.IIterableSource to be precise) with corresponding titles and tokens for its contents. Every now and then a source requires an explicit interface. For zc.sourcefactory the following code snippet helps:

class IMySource(zope.schema.interfaces.IIterableSource):
    """my source"""

class MySource(
    zc.sourcefactory.contextual.BasicContextualSourceFactory):
    """The source factory."""

    class source_class(
        zc.sourcefactory.source.FactoredContextualSource):
        """This class is being instanciated by the factory.

        It *must* be called source_class.
        """
        zope.interface.implements(IMySource)

    def getValues(self, context):
        …

Of course it is also possible to declare the source_class separately from the source factory and reference it. But since its sole purpose is to hold an implements declaration, I’m fine with defining it inline.