union.cms is a content management system which was once developed on Zope 2. It was one of the early adopters of the Five technology aka using Zope 3 components in Zope 2. Now it is one of the proud early adopters of Zope 4 on Python 3. It is used as CMS for large organisations.
In this post we want to present our process how we tackled the migration and briefly discuss occurred issues. Our migration plan looked like the following:
- Port union.cms to Zope 4 while still running on Python 2.7.
- Rollout this version to production – this already has happened in December 2018, see union.cms launched on Zope 4.
- Port the code and the tests to Python 3 while keeping it compatible with Python 2.7 – this was done without branching a dedicated Python 3 branch, instead with a continuous integration system running against Python 2.7 and 3.7 to ensure the possibility of a prompt deployment.
- Have some releases to production during the migration phase – this included also releases with new features developed in parallel.
- Migrate the ZODB based database to be readable by an application server running on Python 3 – Thank you to everyone who contributed to zodbupdate which allowed to have a smooth migration, details see below.
- Run manual tests on a staging system – by ourselves and by the customer to find edge cases not detected by the automatic tests. (There where only a few which where easy to reproduce and to fix.)
- Rollout to production on Python 3 – this was done at the beginning of November 2019.
- Drop the Python 2 support code – this is still open but of low risk.
During the preparation of the project the migration of the Data.fs seemed to be the hardest part. There where no tools in the beginning and the migration had to be done because of the str vs. binary issues between Python 2 and 3. (For details on this topic see Migrate a Zope ZODB Data.fs to Python 3.)
Actually the hardest part was the migration from Zope 2 to Zope 4. There are some internal changes where it is not easy to see what to do to make our own code compatible. Additionally we updated all dependencies to the newest versions which support both Python 2 and 3 to prepare the final switch to Python 3.
The migration of the code to Python 3 was a lot of work. It included to change some dependencies to other packages which have already been ported to Python 3 instead of depending on unmaintained ones. But most of the dependencies were in a usable state. pylint was used on Python 2 to detect code which will cause problems on Python 3. (This requires to use a pylint version older than version 2. We called it using pylint --py3k --disable=no-absolute-import src/** setup.py.) Most parts of the migration could be done automatically using modernize leaving the more trickier ones for the developers.
The migration of the database ran smoothly. The only issue was hidden inside ZCatalog where some index contents were stored as binary but str was expected. This could be solved by creating and running a migration script. (Details see Products.ZCatalog#83.)
The rollout to production went without problems even though databases of more than 10 GByte size had to be migrated. Thankfully it was possible to do the migration offline instead of being forced to do a live migration.
The whole migration project went about two years. We decided for a slower migration with at least some deployments to production to prove the already done steps in a live environment and to allow new features and bug fixes during the migration project. This approach went well, so we can suggest it for other migration projects.
By now union.cms runs live to Zope 4.1 using Python 3.7. It’s time to celebrate that Zope 4 on Python 3 can be used for actual projects in a live environment. 🎉
