union.cms released on Python 3

union.cms is a content management system which was once developed on Zope 2. It was one of the early adopters of the Five technology aka using Zope 3 components in Zope 2. Now it is one of the proud early adopters of Zope 4 on Python 3. It is used as CMS for large organisations.

In this post we want to present our process how we tackled the migration and briefly discuss occurred issues. Our migration plan looked like the following:

  1. Port union.cms to Zope 4 while still running on Python 2.7.
  2. Rollout this version to production – this already has happened in December 2018, see union.cms launched on Zope 4.
  3. Port the code and the tests to Python 3 while keeping it compatible with Python 2.7 – this was done without branching a dedicated Python 3 branch, instead with a continuous integration system running against Python 2.7 and 3.7 to ensure the possibility of a prompt deployment.
  4. Have some releases to production during the migration phase – this included also releases with new features developed in parallel.
  5. Migrate the ZODB based database to be readable by an application server running on Python 3 – Thank you to everyone who contributed to zodbupdate which allowed to have a smooth migration, details see below.
  6. Run manual tests on a staging system – by ourselves and by the customer to find edge cases not detected by the automatic tests. (There where only a few which where easy to reproduce and to fix.)
  7. Rollout to production on Python 3 – this was done at the beginning of November 2019.
  8. Drop the Python 2 support code – this is still open but of low risk.

During the preparation of the project the migration of the Data.fs seemed to be the hardest part. There where no tools in the beginning and the migration had to be done because of the str vs. binary issues between Python 2 and 3. (For details on this topic see Migrate a Zope ZODB Data.fs to Python 3.)

Actually the hardest part was the migration from Zope 2 to Zope 4. There are some internal changes where it is not easy to see what to do to make our own code compatible. Additionally we updated all dependencies to the newest versions which support both Python 2 and 3 to prepare the final switch to Python 3.

The migration of the code to Python 3 was a lot of work. It included to change some dependencies to other packages which have already been ported to Python 3 instead of depending on unmaintained ones. But most of the dependencies were in a usable state. pylint was used on Python 2 to detect code which will cause problems on Python 3. (This requires to use a pylint version older than version 2. We called it using pylint --py3k --disable=no-absolute-import src/** setup.py.) Most parts of the migration could be done automatically using modernize leaving the more trickier ones for the developers.

The migration of the database ran smoothly. The only issue was hidden inside ZCatalog where some index contents were stored as binary but str was expected. This could be solved by creating and running a migration script. (Details see Products.ZCatalog#83.)

The rollout to production went without problems even though databases of more than 10 GByte size had to be migrated. Thankfully it was possible to do the migration offline instead of being forced to do a live migration.

The whole migration project went about two years. We decided for a slower migration with at least some deployments to production to prove the already done steps in a live environment and to allow new features and bug fixes during the migration project. This approach went well, so we can suggest it for other migration projects.

By now union.cms runs live to Zope 4.1 using Python 3.7. It’s time to celebrate that Zope 4 on Python 3 can be used for actual projects in a live environment. 🎉

Undo transactions by truncating ZODB Data.fs

truncate can be used to permanently set back a ZODB Data.fs to a certain point in transaction history.

Sometimes I break the Data.fs of my ZODB in a way that the Zope instance cannot start any more or I want to try again a migration. In such situations it is handy that writing to a Data.fs means extending the file at the end. So the unwanted transaction can be truncated. Normally I use the following steps to do so:

1. Install ZODB in a virtualenv

This is needed to get the script named fstail. If you are already using Python 3, call:
python3.7 -m venv v_zodb

If you are still on Python 2, call:
virtualenv-2.7 v_zodb

Caution: The Python major version (2 or 3) must match the version you are using for the Zope instance.

Install the script into the virtual environment using:
cd v_zodb
bin/pip install ZODB

2. Stop Zope and ZEO

Stop both Zope and (if used) the ZEO server. This is necessary for your cut to get noticed by client and server.

3. Find the position where to cut

Call fstail. With -n you are able to specify the number of transactions to be shown:
bin/fstail -n 20 path/to/Data.fs

fstail returns something like the following. Newer transaction are at the top: (These lines here are only some extracted from a longer output.)

2019-04-24 08:38:44.622984: hash=0b59c10e6eaa947b2ec0538e26d9b4f9128c03cb
user=' admin' description='/storage/58bdea07-666c-11e9-8a63-34363bceb816/edit' length=19180 offset=12296784 (+97)
2019-04-24 08:38:06.823673: hash=3a595fb50b913bad819f0d5bd8d152e06bc695d7
user=' admin' description='/portal/site/add-page-form' length=132830 offset=12121677 (+58)
2019-02-26 10:28:10.856626: hash=5b2b0fbc33b53875b7110f82b2fe1793245c590b
user=' admin' description='/index_html/pt_editAction' length=444 offset=11409587 (+54)

Using the provided information in user and description you should be able to find the transaction from which on newer transactions should be removed. You need the provided value after offset= to do the cut.

In my example above, if /portal/site/add-page-form is the faulty transaction, my cut point is 12121677.

4. 3, 2, 1 … cut

Caution: Every transaction after the cut point (including the one you took the offset from) will get removed by cutting.

Get a truncate tool of your choice. I am using here one of Folkert van Heusden which comes with MacPorts and claims to be command-line compatible with the (Free-)BSD version.

In my example I would call it using:
truncate -s 12121677 path/to/Data.fs

That’s all. Start ZEO and Zope again to be back in transaction history where you have cut.

References

Celebration: Zope 4 final release

TL;DR: Zope 4 beta phase ended, final version released!

After hard, long years of preparation Earl Zope now finally made it to get a permanent license for the Python 3 wonderland: In September 2016 almost 20 people started with the reanimation of Zope at the Zope Resurrection sprint. This marked the beginning of a wonderful journey for Earl Zope himself for the people who helped him. In August 2017 Earl Zope became aware that his Python 2 country will irreversibly be destroyed by 2020. Earl Zope was successfully applying for for a beta permission for the Python 3 wonderland in September 2017. This beta permission has been extended 9 times to give Earl Zope time to become a good citizen in his new home country.

Earl Zope says a big thank you to all who:

  • contributed to the Python 3 migration even before the resurrection sprint
  • wrote bug reports
  • fixed bugs
  • contributed time and/or money for the migration process
  • encouraged the developers
  • tested beta versions or even used them in production

To be welcome in the Python 3 wonderland many nuts had to be cracked:

  • porting of the code of Zope and its dependencies to Python 3
  • rewrite of RestrictedPython from scratch
  • develop a migration strategy for the ZODB contents aka Data.fs
  • polish the user interface of the Zope management interface (ZMI)
  • and many more…

Earl Zope is looking forward to a happy future in the Python 3 wonderland. Currently he did not yet give up his settling in the Python 2 land. This is planned to happen shortly before or after the Python 2 sunset in the beginning of 2020 when the son of Earl Zope IV becomes the new Earl Zope V. See the roadmap for details.

See the documentation how to install Zope. It also documents the migrations steps.

Zope roadmap

Zope 4 is the successor of Zope 2.13 supporting both Python 2.7 and Python 3. After the release of its final version 4.0:

  • Zope 2.13 will drop into “security fixes only” mode. It will stay in this mode as long as there is a supported Plone version using Zope 2.13: The last Plone version using Zope 2.13 is Plone 5.1. Currently the last two major releases of Plone are officially supported. So with the final release of Plone 7 there will be no supported Plone version running on Zope 2.13. (Plone 5.2 will have a final release soon after the Zope 4 final release.)
  • Zope 4 will only be a short-term supported release. It is an intermediate step to ease the transition to Python 3. There will be only a few bugfix releases until Zope 5 is ready. After 4.0.1 is released the work on Zope 5 will start.
  • Zope 5 will only drop support for Python 2 and for APIs and imports which were already marked as deprecated in Zope 4 or even in Zope 2.13. So software which runs on Zope 4 using Python 3 without DeprecationWarnings should run fine on Zope 5, too. A first beta version of Zope 5 could be released by the end of 2019. With the final release of Zope 5, Zope 4 will drop into “security fixes only” mode.

If you have software running on Zope 2.13 with current Python 2.7, the migration plan looks like the following:

  1. Upgrade to Zope 4 on Python 2.7.
  2. Rollout your software on Zope 4 with Python 2.7 to prove your changes in a live environment.
  3. Upgrade the code of your software to Python 3.
  4. Migrate the ZODB to Python 3.
  5. Rollout your software using Zope 4 on Python 3 to prove it in a live environment.
  6. Drop the Python 2 support in your software.
  7. Upgrade to Zope 5 once it is released.

See the documentation how to install Zope. It also documents the migrations steps.

Zope Spring Cleaning: Last minute information

As the beta permission of Earl Zope in Python 3 wonderland was extended in October 2018, gocept invites Zope developers to the upcoming sprint from 08.05. till 10.05.2019 in Halle (Saale), Germany, to continue together on the work, which is still left.

We aim to polish the last dusty spots on Earl Zope for the final permission to Python 3 wonderland aka the final 4.0 release. As Plone and other applications based on Zope have finally found a way to migrate a ZODB Data.fs created with Python 2 to Python 3, the obstacles for this final permit are almost gone.

So if you have questions concerning migrating databases, it is a good time to join or open an issue on GitHub. As many people are working on Zope during these days, the probability of a quick answer is high.

As organizational tool to coordinate the work, we use GitHub projects again, as it allows cross-repository tracking of issues.

Our current schedule:

  • Wednesday
    • 8:15 Breakfast at gocept kitchen
    • 9:00 Welcome at gocept office and start sprinting afterwards
    • 12:30 Lunch
    • 13:30 Happy sprinting
    • between 15:00 and 16:00 coffee break
    • 18:00 Lights out
    • Going to a local pub
  • Thursday:
    • 8:15 Breakfast
    • 9:00 Standup
    • 12:30 Lunch
    • 13:30 Happy sprinting
    • between 15:00 and 16:00 coffee break
    • 17:00 A game of boules if the weather permits it
    • Going to a local pub
  • Friday:
    • 8:15 Breakfast
    • 9:00 Standup
    • 12:30 Lunch
    • 13:30 Happy sprinting
    • 15:00 Closing meeting
    • 16:00 Lights out

Parking: As Saltlabs in located in a pedestrian zone, the availability of parking spots is rather low. Please use one of the parking decks nearby.

One last hint: The location of the sprint is Leipziger Str. 70, Halle (Saale), Germany.

Zope spring cleaning sprint

The last year before the sunset of the Python 2 land has begun. Earl Zope feels relatively well prepared to live in the Python 3 wonderland.

Some issue are still open which are required for the final permission to stay in Python 3 wonderland:

  • test and improve ZODB migration
  • update the documentation
  • fix some glitches in the visual appearance of Earl Zope
  • fix some issues occurring while working together with other residents of the Python 3 wonderland

We invite you, to join forces for three days in May at Saltlabs in Halle and keep on developing good (database) migration stories for various Zope projects, you might have. So bring your Data.fs and let’s port it to Python.

Sprint information

  • Date: Wednesday, 8th until Friday, 10th of May 2019
  • Location: Leipziger Str. 70, Halle (Saale), Germany

Registration

Please join us via Meetup.

union.cms launched on Zope 4

union.cms is a content management system which was once developed on Zope 2. It was one of the early adopters of the Five technology: Using Zope 3 components in Zope 2. Now it is one of the proud early adopters of Zope 4. It is used as CMS for large organisations.

The porting of union.cms to Zope 4 was not that hard because union.cms does not use things from the dirty corners of Zope 2 which have been cleaned up in Zope 4.

Zope 4 is still in its late beta phase, this required more intense testing. There where no critical problems found before the launch. After launch we had an increased  number of ConflictErrors which required a deeper investigation. But even this problem was solvable in a fair amount of time so it does not hurt the editors working daily with the CMS. (Details see Zope#413.) The users even claim it is running faster.

union.cms currently runs live to Zope 4 although it still uses Python 2.7. We are working hard to port it to Python 3. The launch is planned for the end of 2019.

Zope 4 comes with ZODB 5 which seems to be a big step from ZODB 3 which is used in Zope 2. There were no migration steps necessary besides using the current versions of ZODB and ZEO as union.cms still runs on Python 2.7. Upgrading to Python 3 will require a migration of the database contents. This migration is the last blocker before Zope 4 will get a final release.

Now it’s time to celebrate that the beta version of Zope 4 can be used for actual projects in a live environment. 🎉

Beta permission for Earl Zope extended

Earl Zope was hoping to get the final permission for the Python 3 wonderland on the Saltlabs Sprint in Halle last week. He has made good progress in settling down:

  • Zope claims compatibility with the newest Python standards (Python 3.7).
  • Earl Zope now has new cloths for his administrative interface (called ZMI) as shown on the image above.
  • Zope is playing nicer together with Prince Plone who is also migrating to the Python 3 wonderland
  • Documentation got improved and at least partly updated to Zope 4.
  • Zope got some other bug fixes.

But this was not enough for the immigration authorities to give Earl Zope the final permission. The following was missing:

  • a working and proven story how to migrate Earl Zope’s treasures stored the Data.fs

There was good progress towards this goal but it could not be fully reached. So gocept will have the honour to host another sprint maybe in May next year.

The new beta licence has the id 4.0b6 and can be achieved and used as described in the Zope documentation.

Saltlabs Sprint: last minute information

Earl Zope is now nearly settled down in Python 3 wonderland. On the Zope and Plone sprint from Monday, 1st until Friday, 5th of October 2018 in Halle (Saale), Germany we will work towards the final Zope 4 release aka the final permission for the Python 3 wonderland.

We are currently 33 participants for the sprint. So be prepared for a huge sprint with many interesting people. The Saltlabs have a café (called KOFFIJ) we can use, a big meeting room with big display (aka the Thronsaal) and many smaller rooms including the offices of gocept. So there will be enough room to work in bigger and smaller groups.

To keep the organisational overhead low with this amount of participants, we plan to separate in two teams: Zope and Plone. Those teams will organise themselves individually and we will have a short daily meeting after lunch to share the status in a condensed manner with the other team. Direct communication in case of a difficult problem is, of course, always possible.

We reserved up to one hour after the daily meeting for talks and presentations by you about interesting topics around Zope and Plone, successful migration stories, or something else you want to share with the community. So if you have some interesting slides, please bring them with you and register during the week for a slot.

Our current schedule:

  • Sunday
    • 19:00, there is a table reserved at Grober Gottlieb, so if you’ve already arrived and want some company, you are invited to join.
  • Monday
    • 9:00 Breakfast at KOFFIJ (This is the café in the ground floor of Saltlabs aka the window to the left on the picture above.)
    • 10:00 Welcome at KOFFIJ and start sprinting afterwards
    • 12:30 Lunch
    • 13:30 Sprint planning and introduction for all sprinters at Thronsaal
    • between 15:00 and 17:00 coffee break at KOFFIJ
    • 18:00 Lights out
  • All other days:
    • 8:30 Breakfast
    • 9:00 Standup in the team (Zope, Plone)
    • 12:30 Lunch
    • 13:30 Daily meeting at Thronsaal
    • 14:00 (Lightning) Talks at Thronsaal
    • between 15:00 and 17:00 coffee break at KOFFIJ
    • 18:00 Lights out
  • Tuesday:
    • 11:00 till 17:00 Massages, there will be a list to register on Monday
    • 19:00 social evening at Eigenbaukombinat (local hacker space) with pizza, beer and mate
  • Friday:
    • 13:30 Closing meeting with presentations at Thronsaal
    • 17:00 Lights out

If you cannot make it to the Welcome meeting, ask at KOFFIJ for one of the gocept staff to get a personal introduction.

Parking: As Saltlabs in located in a pedestrian zone, the availability of parking spots is rather low. Please use one of the parking decks nearby.

As organizational tool to coordinate the work, we try to use Github projects this time, as it allows cross-repository tracking of issues.

One last hint: The location of the sprint is Leipziger Str. 70, Halle (Saale), Germany.

Goodbye, Forsterstraße. Hello Saltlabs!

After more than 10 years we left our beloved office with its beautiful garden a few weeks ago and moved into “Saltlabs” in the center of Halle – which was created as a workplace for IT companies, featuring a coffee bar, shared infrastructure (kitchen, fast network, meeting rooms with large screens) and will also get a large coworking space next year.

Our new address:

gocept gmbh & co. kg
Leipziger Straße 70
06108 Halle (Saale)
Germany

If you are in Halle, be sure to stop by and have a coffee with us. 🙂