{"id":268,"date":"2012-11-09T18:19:12","date_gmt":"2012-11-09T17:19:12","guid":{"rendered":"http:\/\/blog.gocept.com\/?p=268"},"modified":"2012-12-07T22:09:51","modified_gmt":"2012-12-07T21:09:51","slug":"python-2-and-3-compatible-builds-with-zc-buildout","status":"publish","type":"post","link":"https:\/\/blog.gocept.com\/2012\/11\/09\/python-2-and-3-compatible-builds-with-zc-buildout\/","title":{"rendered":"Python 2 and 3 compatible builds with zc.buildout"},"content":{"rendered":"
\n

Creating a single-source build environment with zc.buildout that works for both Python 2 and 3 is a bit of a hassle. This blog post shows how to do it for a minimal demo project.<\/p>\n

During the sprints at PyCon DE 2012<\/a>, we tried to make the upcoming 1.0 release of the nagiosplugin<\/a> library compatible with both Python 2.7 and Python 3.2. Going for a single code base (without preprocessing steps like 3to2<\/a>) was no too hard. The only thing left was a single-source zc.buildout setup suited for both Python 2.7 and 3.2. It worked out at last, but currently it needs two buildout configurations. This is a little bit kludgy. I hope that things will improve in the near future so that a single-source build environment with zc.buildout will be possible.<\/p>\n

In the following, I will demonstrate the steps with a simple demo project called MultiVersion<\/cite>. It contains nothing more than a single class that is supposed to run under both Python 2 and 3. There is also a unit test to verify that the code works. We use zope.testrunner<\/a> to run the unit tests. The code\u2019s functionality is irrelevant for the examples, so I left it out. You can download the full source<\/a> if you are interested.<\/p>\n

\n

1. Use a recent enough virtualenv<\/h2>\n

Older versions of virtualenv<\/a> are generally not suited since they ship with obsolete releases of distribute<\/cite> and pip<\/cite>. Check if the virtualenv included in your GNU\/Linux distribution is too old. Anything below 1.8 reduces the chance of success, so better install a current virtualenv locally then. Likewise, our bootstrap.py must be recent enough<\/a> to support both Python 2 and 3. The standard bootstrap.py<\/a> from python-distribute.org<\/a> does currently not work with Python 3.<\/p>\n

Now we are ready to create a virtualenv in a fresh source checkout.<\/p>\n

Python 3.2:<\/p>\n

\n
\n
$ virtualenv -p python3.2 .\r\nRunning virtualenv with interpreter \/usr\/bin\/python3.2\r\nNew python executable in .\/bin\/python3.2\r\nInstalling distribute.....done.\r\nInstalling pip.....done.<\/pre>\n<\/div>\n<\/div>\n
Python 2.7:<\/p>\n
\n
\n
$ virtualenv -p python2.7 .\r\nRunning virtualenv with interpreter \/usr\/bin\/python2.7\r\nNew python executable in .\/bin\/python2.7\r\nNot overwriting existing python script .\/bin\/python (you must use .\/bin\/python2.7)\r\nInstalling setuptools.....done.\r\nInstalling pip.....done.<\/pre>\n<\/div>\n<\/div>\n<\/div>\n
\n
2. Running buildout with Python 3.2<\/h2>\n
I will discuss the steps for Python 3.2 first, since main development will concentrate on newer Python versions. After that, I will describe the necessary steps to make the build environment backward compatible.<\/p>\n
To run zc.buildout, we need a buildout.cfg<\/cite> file. I prefer to pin package versions in all projects to ensure reliable builds. As of writing this blog post, there is just an alpha release<\/a> of zc.buildout that supports Python 3.2. Unfortunately, this version of zc.buildout supports Python 3.2 only, so don\u2019t try this with Python 3.3.<\/p>\n
My basic buildout.cfg<\/cite> looks like this:<\/p>\n
\n
[buildout]\r\nallow-picked-versions = false\r\ndevelop = .\r\nnewest = false\r\npackage = multiversion\r\nparts = multiversion test\r\nversions = versions\r\n\r\n[versions]\r\ndistribute = 0.6.28\r\nz3c.recipe.scripts = 1.0.1\r\nzc.buildout = 2.0.0a2\r\nzc.recipe.egg = 2.0.0a2\r\nzc.recipe.testrunner = 1.4.0\r\nzope.exceptions = 4.0.1\r\nzope.interface = 4.0.1\r\nzope.testrunner = 4.0.4\r\n\r\n[multiversion]\r\nrecipe = zc.recipe.egg\r\neggs = ${buildout:package}\r\ninterpreter = py\r\n\r\n[test]\r\nrecipe = zc.recipe.testrunner\r\neggs = ${buildout:package}\r\ndefaults = ['--auto-color']<\/pre>\n<\/div>\n
In my experience, it is best to pin distutils to exactly the same version that is included in virtualenv\u2019s support files. While differing versions are possible, they may trigger hard to find bugs since it is not always clear which version is used is which step.<\/p>\n
I use the Python interpreter from my virtualenv\u2019s bin<\/cite> directory while creating the buildout executable. This saves me from using activate\/deactivate scripts which are slightly cumbersome in my opinion.<\/p>\n
\n
\n
$ bin\/python3.2 bootstrap.py\r\nCreating directory 'blog-python-2-3\/parts'.\r\nCreating directory 'blog-python-2-3\/develop-eggs'.\r\nGenerated script 'blog-python-2-3\/bin\/buildout'.\r\n\r\n$ bin\/buildout\r\nDevelop: 'blog-python-2-3\/.'\r\nInstalling multiversion.\r\nGenerated interpreter 'blog-python-2-3\/bin\/py'.\r\nInstalling test.\r\nGenerated script 'blog-python-2-3\/bin\/test'.<\/pre>\n<\/div>\n<\/div>\n
Now we have a working build for Python 3.2:<\/p>\n
\n
\n
$ bin\/test\r\nRunning zope.testrunner.layer.UnitTests tests:\r\n  Set up zope.testrunner.layer.UnitTests in 0.000 seconds.\r\n  Ran 1 tests with 0 failures and 0 errors in 0.002 seconds.\r\nTearing down left over layers:\r\n  Tear down zope.testrunner.layer.UnitTests in 0.000 seconds.<\/pre>\n<\/div>\n<\/div>\n<\/div>\n
\n
3. Running buildout with Python 2.7<\/h2>\n
Unfortunately, the current zc.buildout alpha release does not work with anything except Python 3.2. Running bootstrap.py fails:<\/p>\n
\n
$ bin\/python2.7 bootstrap.py\r\nGetting distribution for 'zc.buildout==2.0.0a2'.\r\nWhile:\r\n  Bootstrapping.\r\n  Getting distribution for 'zc.buildout==2.0.0a2'.\r\nError: Couldn't find a distribution for 'zc.buildout==2.0.0a2'.<\/pre>\n<\/div>\n
There is no single zc.buildout distribution that fits both Python 2.7 and 3.2. To get around this, I need to create a special-case buildout.cfg<\/cite> that changes version pinnings for incompatible packages. Besides zc.buildout, zc.recipe.egg needs different versions for Python 2.7 and 3.2 as well.<\/p>\n
I create buildout-2.x.cfg<\/cite> (slightly grumbling):<\/p>\n
\n
[buildout]\r\nextends = buildout.cfg\r\n\r\n[versions]\r\nzc.buildout = 1.6.3\r\nzc.recipe.egg = 1.3.2<\/pre>\n<\/div>\n
This one does the job when used with both bootstrap and buildout:<\/p>\n
\n
\n
$ bin\/python2.7 bootstrap.py -c buildout-2.x.cfg\r\nGenerated script 'blog-python-2-3\/bin\/buildout'.\r\n\r\n$ bin\/buildout -c buildout-2.x.cfg\r\nDevelop: 'blog-python-2-3\/.'\r\nInstalling multiversion.\r\nGenerated interpreter 'blog-python-2-3\/bin\/py'.\r\nInstalling test.\r\nGenerated script 'blog-python-2-3\/bin\/test'.<\/pre>\n<\/div>\n<\/div>\n
We now have a build environment that builds single-source code for both Python 2.7 and 3.2 using zc.buildout. Of course, this technique could be extended to support even more versions. But I hope that the incompatible packages will be updated in the near future so that the need for special-case buildout.cfg<\/cite> files will go away. What seems to be most missing: a release of zc.buildout that supports all major Python versions.<\/p>\n<\/div>\n
\n
TL;DR<\/h2>\n