union.cms released on Python 3

union.cms is a content management system which was once developed on Zope 2. It was one of the early adopters of the Five technology aka using Zope 3 components in Zope 2. Now it is one of the proud early adopters of Zope 4 on Python 3. It is used as CMS for large organisations.

In this post we want to present our process how we tackled the migration and briefly discuss occurred issues. Our migration plan looked like the following:

  1. Port union.cms to Zope 4 while still running on Python 2.7.
  2. Rollout this version to production – this already has happened in December 2018, see union.cms launched on Zope 4.
  3. Port the code and the tests to Python 3 while keeping it compatible with Python 2.7 – this was done without branching a dedicated Python 3 branch, instead with a continuous integration system running against Python 2.7 and 3.7 to ensure the possibility of a prompt deployment.
  4. Have some releases to production during the migration phase – this included also releases with new features developed in parallel.
  5. Migrate the ZODB based database to be readable by an application server running on Python 3 – Thank you to everyone who contributed to zodbupdate which allowed to have a smooth migration, details see below.
  6. Run manual tests on a staging system – by ourselves and by the customer to find edge cases not detected by the automatic tests. (There where only a few which where easy to reproduce and to fix.)
  7. Rollout to production on Python 3 – this was done at the beginning of November 2019.
  8. Drop the Python 2 support code – this is still open but of low risk.

During the preparation of the project the migration of the Data.fs seemed to be the hardest part. There where no tools in the beginning and the migration had to be done because of the str vs. binary issues between Python 2 and 3. (For details on this topic see Migrate a Zope ZODB Data.fs to Python 3.)

Actually the hardest part was the migration from Zope 2 to Zope 4. There are some internal changes where it is not easy to see what to do to make our own code compatible. Additionally we updated all dependencies to the newest versions which support both Python 2 and 3 to prepare the final switch to Python 3.

The migration of the code to Python 3 was a lot of work. It included to change some dependencies to other packages which have already been ported to Python 3 instead of depending on unmaintained ones. But most of the dependencies were in a usable state. pylint was used on Python 2 to detect code which will cause problems on Python 3. (This requires to use a pylint version older than version 2. We called it using pylint --py3k --disable=no-absolute-import src/** setup.py.) Most parts of the migration could be done automatically using modernize leaving the more trickier ones for the developers.

The migration of the database ran smoothly. The only issue was hidden inside ZCatalog where some index contents were stored as binary but str was expected. This could be solved by creating and running a migration script. (Details see Products.ZCatalog#83.)

The rollout to production went without problems even though databases of more than 10 GByte size had to be migrated. Thankfully it was possible to do the migration offline instead of being forced to do a live migration.

The whole migration project went about two years. We decided for a slower migration with at least some deployments to production to prove the already done steps in a live environment and to allow new features and bug fixes during the migration project. This approach went well, so we can suggest it for other migration projects.

By now union.cms runs live to Zope 4.1 using Python 3.7. It’s time to celebrate that Zope 4 on Python 3 can be used for actual projects in a live environment. 🎉

union.cms launched on Zope 4

union.cms is a content management system which was once developed on Zope 2. It was one of the early adopters of the Five technology: Using Zope 3 components in Zope 2. Now it is one of the proud early adopters of Zope 4. It is used as CMS for large organisations.

The porting of union.cms to Zope 4 was not that hard because union.cms does not use things from the dirty corners of Zope 2 which have been cleaned up in Zope 4.

Zope 4 is still in its late beta phase, this required more intense testing. There where no critical problems found before the launch. After launch we had an increased  number of ConflictErrors which required a deeper investigation. But even this problem was solvable in a fair amount of time so it does not hurt the editors working daily with the CMS. (Details see Zope#413.) The users even claim it is running faster.

union.cms currently runs live to Zope 4 although it still uses Python 2.7. We are working hard to port it to Python 3. The launch is planned for the end of 2019.

Zope 4 comes with ZODB 5 which seems to be a big step from ZODB 3 which is used in Zope 2. There were no migration steps necessary besides using the current versions of ZODB and ZEO as union.cms still runs on Python 2.7. Upgrading to Python 3 will require a migration of the database contents. This migration is the last blocker before Zope 4 will get a final release.

Now it’s time to celebrate that the beta version of Zope 4 can be used for actual projects in a live environment. 🎉