Author: Michael Howitz

I am a software developer at gocept in Halle (Saale). To develop software, I mainly use Python, Zope, ZTK and Django.

Ready, set, sprint: Earl Zope invites again

Sprint with Earl Zope at September, 21st and 22nd of 2023.

Earl Zope is already a long time inhabitant of the Python 3 wonderland. Living here for some years with the help of his friends, he sometimes looks back to the “good old times” but mostly enjoys his current live with all its surprises and unexpected turns.

Gocept invites you to sprint for Earl Zope’s future well-being: 21st till 22nd of September 2023 in the SaltLabs of Halle, Germany. Participants of the sprint are also invited to join the gocept-23 party at 23rd of September 2023.

This sprint will the last one hosted by gocept as the company will go new ways by the end of the year. To join please register via meetup.com. (After registering for the sprint, you’ll get the party invitation.)

Sprint with Earl Zope in April of 2022

Sprint with Earl Zope at April, 13th 2022 towards Python 3.11 compatibility.

Earl Zope – once in a while – needs some help to feel well again in the ever changing Python 3 wonderland.

Please join a one-day-sprint at April, 13th 2022 organized by gocept to make Earl Zope more comfortable with the current Python 3.10 and the upcoming Python 3.11.

Especially the new upcoming version has some bitter surprises for Earl Zope: The C-code of some of his dependencies will no longer compile without modifications. An example dependency is already ported, so it should be possible to port other ones by the monkey-see-monkey-do principle.

The sprint will be a pure online-sprint. If you want to join please register via meetup.com.

Zope April Sprint 2021 report

Earl Zope invited to a sprint and nearly 10 people were joining to work on several aspects in Earl Zope’s ecosystem:

  • fix the ZEO tests on Python 3.8+
  • add a “delete all” button for Products.BTreeFolder2
  • fix DeprecationWarnings in Products.CMFCore and zope.sqlalchemy
  • migrate CI system of some packages from TravisCI to GitHub Actions
  • document the GitHub teams of the zopefoundation organization
  • …

By the end of the day the sprinters were joined by Paul Everitt and Jim Fulton to hear some stories of the former glory of Earl Zope and to discuss the current status and possibilities.

Earl Zope thanks all the people who joined the sprint and made it possible that he can feel a bit more comfortable in the Python 3 wonderland.

There is another one-day-sprint upcoming at Friday, 2nd of July 2021. Feel free to join via meetup.com. Earl Zope is looking forward to meet you.

Earl Zope invites to sprint

Two Zope sprints at 23rd of April and
2nd of July 2021.

Already nearly two years went by after Earl Zope got his permanent permit to live in the Python 3 wonderland. He enjoys his new life, the many new friends he made and the old ones who also moved to the Python 3 wonderland. Life could be so happy and so easy but there are always some disruptions appearing and shadows of the past showing up:

  • the automatic verification for Earl Zope and all of his staff broke – aka the continuous integration system for Zope and its dependencies has to be changed from TravisCI to GitHub Actions
  • the laws, regulations and instructions have to be adapted to the Python 3 wonderland standards – aka still large parts of the documentation are outdated: they are describing Zope 2, and should get an update to match Zope 5
  • Python 3 wonderland rapidly changes and develops: each year there will be changes Earl Zope has to handle – aka Python 3.10 is in its late alpha versions now and it is already known that there are changes in Zope’s dependencies necessary to support that new version

So Earl Zope kindly asks for help with this troubles.– Or should he call them challenges? He invites to a two short sprints:

Both sprints will be remote-only due to the unclear pandemic situation. On the other hand it allows participants to save time and money otherwise needed for travel.

To participate, please sign up using Meetup: The dates above are links to the Meetup events.

Earl Zope looks forward to these days and thanks in advance everyone who will be contributing to the success.

Presentation: We have nearly one million lines of Python 2 code in production — and now?

Still running Python 2 code in production is like steering a ship without radar in thick fog: You don’t know, which obstacle you will hit next. But there are ways to see the sun again – even for large code bases. This presentation contains a discussion of the possible ways and a success story.

I have been giving the presentation at

  • Python Web Conference 2020 and
  • EuroPython 2020.

If you did not manage to see the presentation at one of these conferences, you will find now:

Maybe this helps you with your next migration project to the Python 3 wonderland. If you have questions – we are here to help with such migration projects.

We have nearly one million lines of Python 2 code in production – and now?

How to successfully migrate a Python 2 project to Python 3.

Still running Python 2 code in production is like steering a ship without radar in thick fog: You don’t know, which obstacle you will hit next. But there are ways to see the sun again even for large code bases.

I am Michael Howitz and I am going to present a talk at Python Web Conf 2020 on this topic.

In this talk I will show possible approaches for a Python 3 migration of existing Python code running on Python 2.7. I will evaluate which of these approaches seems suitable for a large code base and which principles should be paid attention to to increase the probability of a successful migration project.

I am going use union.cms (a content management system used by German trade unions) to show the story of a successful migration of a project with nearly one million lines of Python code. The story will include the obstacles we encountered and what we learned on the way.

I believe that most of the discussed ideas in the talk and experience can also be used unchanged in smaller projects as the presented principles are very much the same.

The talk is scheduled at 1:00 – 1:45 pm ET (UTC -4h) on Friday, June 19th. (That’s 7:00 pm in MEST (UTC+2)).

To see the talk live on your device, join Python Web Conf 2020 now.

Zope May sprint goes remote

Earl Zope was inviting to the Zope May sprint, from 13th till 15th of May, 2020. He was hoping for many volunteers to come. Due to restrictions to prevent spreading of COVID-19 (Corona) this sprint is going to be remote-only. – By now all of the organizers and their families are fine, so you do not have to worry about us.

This remote-only sprint will be an experiment. We already gathered some experience with remote work, communication and planning. But we neither participated at nor organized a remote-only sprint.

We will prepare channels for communication and collaboration for this sprint. Stay tuned for details. Earl Zope will appreciate your help.

As even a remote-only sprint needs preparation, please join via Meetup, so we are able to see, who we can count on to help Earl Zope in Python wonderland.

Zope May Sprint

Earl Zope has settled down for a good while in Python 3 wonderland. He made friends with the inhabitants and other immigrants. He enjoys his new live.

The sunset of his original homelands took place as predicted by the beginning of January 2020. As Earl Zope was well prepared this was no longer a frightening date for him.

But even living in Python 3 wonderland is not only joy and relaxing. The Python 3 wonderland changes in a more rapid speed than the Python 2 land ever had before: Each year a new policy has to be fulfilled (aka new Python version release). Additionally it is time to drop the last connections to the old Python 2 land to ease the transformation in Python 3 wonderland to make developers and consumers happy.

Earl Zope is grateful for all the help he already gained: There where several Zope 4 releases and a first Zope 5 alpha version was just released. Even though Earl Zope still needs your help to:

  • prepare dependencies to ease transition to new Python versions (aka make repositories more uniform to ease updating to new Python versions.)
  • drop Python 2 support in repositories of dependencies
  • support and test with newer Python 3 versions (aka current 3.9 alpha)
  • improve and update the documentation

You are invited to the “Zope May sprint” located in Halle/Saale, 🇩🇪 from 13th till 15th of May 2020 hosted by gocept. In order to coordinate the participation for this sprint, we ask you to join us on Meetup. We can then coordinate the catering and requirements for space.

Update: This sprint will be held als a remote-only sprint. Details see Zope May sprint goes remote.

union.cms released on Python 3

union.cms is a content management system which was once developed on Zope 2. It was one of the early adopters of the Five technology aka using Zope 3 components in Zope 2. Now it is one of the proud early adopters of Zope 4 on Python 3. It is used as CMS for large organisations.

In this post we want to present our process how we tackled the migration and briefly discuss occurred issues. Our migration plan looked like the following:

  1. Port union.cms to Zope 4 while still running on Python 2.7.
  2. Rollout this version to production – this already has happened in December 2018, see union.cms launched on Zope 4.
  3. Port the code and the tests to Python 3 while keeping it compatible with Python 2.7 – this was done without branching a dedicated Python 3 branch, instead with a continuous integration system running against Python 2.7 and 3.7 to ensure the possibility of a prompt deployment.
  4. Have some releases to production during the migration phase – this included also releases with new features developed in parallel.
  5. Migrate the ZODB based database to be readable by an application server running on Python 3 – Thank you to everyone who contributed to zodbupdate which allowed to have a smooth migration, details see below.
  6. Run manual tests on a staging system – by ourselves and by the customer to find edge cases not detected by the automatic tests. (There where only a few which where easy to reproduce and to fix.)
  7. Rollout to production on Python 3 – this was done at the beginning of November 2019.
  8. Drop the Python 2 support code – this is still open but of low risk.

During the preparation of the project the migration of the Data.fs seemed to be the hardest part. There where no tools in the beginning and the migration had to be done because of the str vs. binary issues between Python 2 and 3. (For details on this topic see Migrate a Zope ZODB Data.fs to Python 3.)

Actually the hardest part was the migration from Zope 2 to Zope 4. There are some internal changes where it is not easy to see what to do to make our own code compatible. Additionally we updated all dependencies to the newest versions which support both Python 2 and 3 to prepare the final switch to Python 3.

The migration of the code to Python 3 was a lot of work. It included to change some dependencies to other packages which have already been ported to Python 3 instead of depending on unmaintained ones. But most of the dependencies were in a usable state. pylint was used on Python 2 to detect code which will cause problems on Python 3. (This requires to use a pylint version older than version 2. We called it using pylint --py3k --disable=no-absolute-import src/** setup.py.) Most parts of the migration could be done automatically using modernize leaving the more trickier ones for the developers.

The migration of the database ran smoothly. The only issue was hidden inside ZCatalog where some index contents were stored as binary but str was expected. This could be solved by creating and running a migration script. (Details see Products.ZCatalog#83.)

The rollout to production went without problems even though databases of more than 10 GByte size had to be migrated. Thankfully it was possible to do the migration offline instead of being forced to do a live migration.

The whole migration project went about two years. We decided for a slower migration with at least some deployments to production to prove the already done steps in a live environment and to allow new features and bug fixes during the migration project. This approach went well, so we can suggest it for other migration projects.

By now union.cms runs live to Zope 4.1 using Python 3.7. It’s time to celebrate that Zope 4 on Python 3 can be used for actual projects in a live environment. 🎉

Undo transactions by truncating ZODB Data.fs

truncate can be used to permanently set back a ZODB Data.fs to a certain point in transaction history.

Sometimes I break the Data.fs of my ZODB in a way that the Zope instance cannot start any more or I want to try again a migration. In such situations it is handy that writing to a Data.fs means extending the file at the end. So the unwanted transaction can be truncated. Normally I use the following steps to do so:

1. Install ZODB in a virtualenv

This is needed to get the script named fstail. If you are already using Python 3, call:
python3.7 -m venv v_zodb

If you are still on Python 2, call:
virtualenv-2.7 v_zodb

Caution: The Python major version (2 or 3) must match the version you are using for the Zope instance.

Install the script into the virtual environment using:
cd v_zodb
bin/pip install ZODB

2. Stop Zope and ZEO

Stop both Zope and (if used) the ZEO server. This is necessary for your cut to get noticed by client and server.

3. Find the position where to cut

Call fstail. With -n you are able to specify the number of transactions to be shown:
bin/fstail -n 20 path/to/Data.fs

fstail returns something like the following. Newer transaction are at the top: (These lines here are only some extracted from a longer output.)

2019-04-24 08:38:44.622984: hash=0b59c10e6eaa947b2ec0538e26d9b4f9128c03cb
user=' admin' description='/storage/58bdea07-666c-11e9-8a63-34363bceb816/edit' length=19180 offset=12296784 (+97)
2019-04-24 08:38:06.823673: hash=3a595fb50b913bad819f0d5bd8d152e06bc695d7
user=' admin' description='/portal/site/add-page-form' length=132830 offset=12121677 (+58)
2019-02-26 10:28:10.856626: hash=5b2b0fbc33b53875b7110f82b2fe1793245c590b
user=' admin' description='/index_html/pt_editAction' length=444 offset=11409587 (+54)

Using the provided information in user and description you should be able to find the transaction from which on newer transactions should be removed. You need the provided value after offset= to do the cut.

In my example above, if /portal/site/add-page-form is the faulty transaction, my cut point is 12121677.

4. 3, 2, 1 … cut

Caution: Every transaction after the cut point (including the one you took the offset from) will get removed by cutting.

Get a truncate tool of your choice. I am using here one of Folkert van Heusden which comes with MacPorts and claims to be command-line compatible with the (Free-)BSD version.

In my example I would call it using:
truncate -s 12121677 path/to/Data.fs

That’s all. Start ZEO and Zope again to be back in transaction history where you have cut.

References