Tag: python3

Presentation: We have nearly one million lines of Python 2 code in production — and now?

Still running Python 2 code in production is like steering a ship without radar in thick fog: You don’t know, which obstacle you will hit next. But there are ways to see the sun again – even for large code bases. This presentation contains a discussion of the possible ways and a success story.

I have been giving the presentation at

  • Python Web Conference 2020 and
  • EuroPython 2020.

If you did not manage to see the presentation at one of these conferences, you will find now:

Maybe this helps you with your next migration project to the Python 3 wonderland. If you have questions – we are here to help with such migration projects.

We have nearly one million lines of Python 2 code in production – and now?

How to successfully migrate a Python 2 project to Python 3.

Still running Python 2 code in production is like steering a ship without radar in thick fog: You don’t know, which obstacle you will hit next. But there are ways to see the sun again even for large code bases.

I am Michael Howitz and I am going to present a talk at Python Web Conf 2020 on this topic.

In this talk I will show possible approaches for a Python 3 migration of existing Python code running on Python 2.7. I will evaluate which of these approaches seems suitable for a large code base and which principles should be paid attention to to increase the probability of a successful migration project.

I am going use union.cms (a content management system used by German trade unions) to show the story of a successful migration of a project with nearly one million lines of Python code. The story will include the obstacles we encountered and what we learned on the way.

I believe that most of the discussed ideas in the talk and experience can also be used unchanged in smaller projects as the presented principles are very much the same.

The talk is scheduled at 1:00 – 1:45 pm ET (UTC -4h) on Friday, June 19th. (That’s 7:00 pm in MEST (UTC+2)).

To see the talk live on your device, join Python Web Conf 2020 now.

Zope May Sprint

Earl Zope has settled down for a good while in Python 3 wonderland. He made friends with the inhabitants and other immigrants. He enjoys his new live.

The sunset of his original homelands took place as predicted by the beginning of January 2020. As Earl Zope was well prepared this was no longer a frightening date for him.

But even living in Python 3 wonderland is not only joy and relaxing. The Python 3 wonderland changes in a more rapid speed than the Python 2 land ever had before: Each year a new policy has to be fulfilled (aka new Python version release). Additionally it is time to drop the last connections to the old Python 2 land to ease the transformation in Python 3 wonderland to make developers and consumers happy.

Earl Zope is grateful for all the help he already gained: There where several Zope 4 releases and a first Zope 5 alpha version was just released. Even though Earl Zope still needs your help to:

  • prepare dependencies to ease transition to new Python versions (aka make repositories more uniform to ease updating to new Python versions.)
  • drop Python 2 support in repositories of dependencies
  • support and test with newer Python 3 versions (aka current 3.9 alpha)
  • improve and update the documentation

You are invited to the “Zope May sprint” located in Halle/Saale, 🇩🇪 from 13th till 15th of May 2020 hosted by gocept. In order to coordinate the participation for this sprint, we ask you to join us on Meetup. We can then coordinate the catering and requirements for space.

Update: This sprint will be held als a remote-only sprint. Details see Zope May sprint goes remote.

union.cms released on Python 3

union.cms is a content management system which was once developed on Zope 2. It was one of the early adopters of the Five technology aka using Zope 3 components in Zope 2. Now it is one of the proud early adopters of Zope 4 on Python 3. It is used as CMS for large organisations.

In this post we want to present our process how we tackled the migration and briefly discuss occurred issues. Our migration plan looked like the following:

  1. Port union.cms to Zope 4 while still running on Python 2.7.
  2. Rollout this version to production – this already has happened in December 2018, see union.cms launched on Zope 4.
  3. Port the code and the tests to Python 3 while keeping it compatible with Python 2.7 – this was done without branching a dedicated Python 3 branch, instead with a continuous integration system running against Python 2.7 and 3.7 to ensure the possibility of a prompt deployment.
  4. Have some releases to production during the migration phase – this included also releases with new features developed in parallel.
  5. Migrate the ZODB based database to be readable by an application server running on Python 3 – Thank you to everyone who contributed to zodbupdate which allowed to have a smooth migration, details see below.
  6. Run manual tests on a staging system – by ourselves and by the customer to find edge cases not detected by the automatic tests. (There where only a few which where easy to reproduce and to fix.)
  7. Rollout to production on Python 3 – this was done at the beginning of November 2019.
  8. Drop the Python 2 support code – this is still open but of low risk.

During the preparation of the project the migration of the Data.fs seemed to be the hardest part. There where no tools in the beginning and the migration had to be done because of the str vs. binary issues between Python 2 and 3. (For details on this topic see Migrate a Zope ZODB Data.fs to Python 3.)

Actually the hardest part was the migration from Zope 2 to Zope 4. There are some internal changes where it is not easy to see what to do to make our own code compatible. Additionally we updated all dependencies to the newest versions which support both Python 2 and 3 to prepare the final switch to Python 3.

The migration of the code to Python 3 was a lot of work. It included to change some dependencies to other packages which have already been ported to Python 3 instead of depending on unmaintained ones. But most of the dependencies were in a usable state. pylint was used on Python 2 to detect code which will cause problems on Python 3. (This requires to use a pylint version older than version 2. We called it using pylint --py3k --disable=no-absolute-import src/** setup.py.) Most parts of the migration could be done automatically using modernize leaving the more trickier ones for the developers.

The migration of the database ran smoothly. The only issue was hidden inside ZCatalog where some index contents were stored as binary but str was expected. This could be solved by creating and running a migration script. (Details see Products.ZCatalog#83.)

The rollout to production went without problems even though databases of more than 10 GByte size had to be migrated. Thankfully it was possible to do the migration offline instead of being forced to do a live migration.

The whole migration project went about two years. We decided for a slower migration with at least some deployments to production to prove the already done steps in a live environment and to allow new features and bug fixes during the migration project. This approach went well, so we can suggest it for other migration projects.

By now union.cms runs live to Zope 4.1 using Python 3.7. It’s time to celebrate that Zope 4 on Python 3 can be used for actual projects in a live environment. 🎉

Celebration: Zope 4 final release

TL;DR: Zope 4 beta phase ended, final version released!

After hard, long years of preparation Earl Zope now finally made it to get a permanent license for the Python 3 wonderland: In September 2016 almost 20 people started with the reanimation of Zope at the Zope Resurrection sprint. This marked the beginning of a wonderful journey for Earl Zope himself for the people who helped him. In August 2017 Earl Zope became aware that his Python 2 country will irreversibly be destroyed by 2020. Earl Zope was successfully applying for for a beta permission for the Python 3 wonderland in September 2017. This beta permission has been extended 9 times to give Earl Zope time to become a good citizen in his new home country.

Earl Zope says a big thank you to all who:

  • contributed to the Python 3 migration even before the resurrection sprint
  • wrote bug reports
  • fixed bugs
  • contributed time and/or money for the migration process
  • encouraged the developers
  • tested beta versions or even used them in production

To be welcome in the Python 3 wonderland many nuts had to be cracked:

  • porting of the code of Zope and its dependencies to Python 3
  • rewrite of RestrictedPython from scratch
  • develop a migration strategy for the ZODB contents aka Data.fs
  • polish the user interface of the Zope management interface (ZMI)
  • and many more…

Earl Zope is looking forward to a happy future in the Python 3 wonderland. Currently he did not yet give up his settling in the Python 2 land. This is planned to happen shortly before or after the Python 2 sunset in the beginning of 2020 when the son of Earl Zope IV becomes the new Earl Zope V. See the roadmap for details.

See the documentation how to install Zope. It also documents the migrations steps.

Zope spring cleaning sprint

The last year before the sunset of the Python 2 land has begun. Earl Zope feels relatively well prepared to live in the Python 3 wonderland.

Some issue are still open which are required for the final permission to stay in Python 3 wonderland:

  • test and improve ZODB migration
  • update the documentation
  • fix some glitches in the visual appearance of Earl Zope
  • fix some issues occurring while working together with other residents of the Python 3 wonderland

We invite you, to join forces for three days in May at Saltlabs in Halle and keep on developing good (database) migration stories for various Zope projects, you might have. So bring your Data.fs and let’s port it to Python.

Sprint information

  • Date: Wednesday, 8th until Friday, 10th of May 2019
  • Location: Leipziger Str. 70, Halle (Saale), Germany

Registration

Please join us via Meetup.

Beta permission for Earl Zope extended

Earl Zope was hoping to get the final permission for the Python 3 wonderland on the Saltlabs Sprint in Halle last week. He has made good progress in settling down:

  • Zope claims compatibility with the newest Python standards (Python 3.7).
  • Earl Zope now has new cloths for his administrative interface (called ZMI) as shown on the image above.
  • Zope is playing nicer together with Prince Plone who is also migrating to the Python 3 wonderland
  • Documentation got improved and at least partly updated to Zope 4.
  • Zope got some other bug fixes.

But this was not enough for the immigration authorities to give Earl Zope the final permission. The following was missing:

  • a working and proven story how to migrate Earl Zope’s treasures stored the Data.fs

There was good progress towards this goal but it could not be fully reached. So gocept will have the honour to host another sprint maybe in May next year.

The new beta licence has the id 4.0b6 and can be achieved and used as described in the Zope documentation.

Saltlabs Sprint: Zope and Plone sprint in a new location

After Earl Zope II is now nearly relocated to the Python 3 wonderland, gocept will move to a new head quarter in the next months. This is the right time to celebrate with a new sprint, as we have now even more space for sprinters. The new location is called the “Saltlabs”, a place for IT companies in Halle (Saale), Germany.

Sprint information

  • Date: Monday, 1st until Friday, 5th of October 2018
  • Location: Leipziger Str. 70, Halle (Saale), Germany

Sprint topics

This sprint has three main topics:

Create a final Zope 4 release

Before releasing a final version of Zope 4 we want to resolve about at least 40 issues: Some bugs have to be fixed,  some functions have to be polished and documentation has to be written resp. reviewed. On the other hand there is the re-brush of the ZMI using Bootstrap which should be completed beforehand, as it modernizes the ZMI and allows for easier customisation, but might also be backwards incompatible with certain test suites. There is an Etherpad to write down ideas, tasks, wishes and work proposals, which are not currently covered by the issue tracker.

Port Plone to Python 3

The following tasks are currently open and can be fixed at the sprint:

  • successfully run all Plone tests and even the robotframework tests on Python 3
  • Zope 4 lost the WebDAV support: find resp. create a replacement
  • document the WSGI setup and test it in a production ready environment
  • port as many as possible add-ons to Python 3 (e.g. Mosaic and Easyform)
  • work on the Migration of ZODB contents (Data.fs) to Python 3
  • improve the test setup with tox.
  • start to support Python 3.7

Polish Plone 5.2

The upcoming Plone 5.2 release will appreciate some love and care at the following items:

  • new navigation with dropdown and better performance
  • Barceloneta theme: ease the customisation and improve responsiveness
  • parallelise the tests so they run faster
  • remove Archetypes and other obsolete packages

See also the list of topics on plone.org for this sprint.

Organisational Remarks

In order to coordinate the participation for this sprint, we ask you to join us on Meetup. We can then coordinate the catering and requirements for space.

As this sprint will be running longer than usual (five days), it is also possible to join only for a part of the week. As October 3rd is the national holiday, we are trying to organise some social event for those who are interested in having a small break.

For a better overview, please indicate your participation also on this doodle poll.

 

Migrate a Zope ZODB Data.fs to Python 3

TL;DR Use zodbupdate.

Problem

A ZODB Data.fs which was created under Python 2 cannot be opened under Python 3. This is prevented by using a different magic code in the first bytes of the file. This is done on purpose because str has a different meaning for the two Python versions: Under Python 2 a str is a container for characters with an arbitrary encoding (aka bytes​). Python 3 knows str as a text datatype which was called unicode in Python 2. Trying to load a str object in Python 3 which actually contains binary data will fail. It has to be bytes, but bytes is an alias for str  in Python 2 which means Python 2 replaces bytes  with str making is impossible to give Python 3 the class it expects for binary data. A Python 2 str  with an arbitrary encoding will break, too.

Solution

The Data.fs has to be migrated: each str  which actually contains bytes has to be converted into a zodbpickle.binary object which deserialises as bytes under Python 3. The str objects actually containing text have to be decoded to unicode. There are currently two tools which claim that they are able to do such a migration:

  • zodb.py3migrate was already written at Berlin Strategic sprint in 2016, but it was never able to prove that it can do what it claims: At the time when it was written there was no Zope which could run on Python 3. Now as we have Zope 4 running on Python 3 it does not seem to do its conversion job quite well: I was able to migrate a toy database but had to catch an unpickling error.
  • zodbupdate was enriched by a Python 3 migration. A big thank you to Sylvain Viollon and the developers at Minddistrict! It has proven its claims! At the Zope 4 welcome sprint I was able to migrate a Data.fs created on Zope 2.13 running on Python 2 to Zope 4 running on Python 3.

Steps

  1. Migrate your Zope application to Zope 4. (zodbupdate  requires at least ZODB 4 which is not the default ZODB version of Zope 2.13) — For my toy database containing only a file object and an image this was no problem. Zope 4  is starting with such a database. It might show some broken objects because Zope no longer depends on some previous core packages like Products.Sessions. If your application needs those packages you should add them to your Zope environment.
  2. ​zodbupdate has to be installed into the Zope 4 environment so it can access the Python classes. (It has to read the pickles in the ZODB.)
  3. There needs to be an entry_point in setup.py for each package which contains persistent Python classes. The entry point has to be named "zodbupdate.decode" and needs to point to a dictionary mapping paths to  str attributes to a conversion (bytes resp. a specific encoding). For Details see the migration documentation of zodbupdate. I prepared a branch of Zope 4 which contains this configuration dictionary for OFS.Image and OFS.File, see zopefoundation/Zope#285.
  4. Run zodbupdate --pack --convert-py3 on the Data.fs using Python 2.
  5. Copy the Data.fs over to the Zope 4 instance running on Python 3. Data.fs.index will be discarded at the first start. (There is an error message telling that it cannot be read.)
  6. Enjoy the contents of the Data.fs running on Python 3.

Conclusion

It is possible (proven for a toy database) to migrate a Data.fs from Zope 2.13  (Python 2) to Zope 4 (Python 3).

zodbupdate is the way to go. Although it cannot do the migration completely autonomously the developers of Python packages can provide migration configuration in their packages which can be used in the migration step so the configuration has only to be written once.

zodb.py3migrate has an analysis step which shows the attribute names where the str objects are stored. (This could be added to zodbupdate, so do not expect that there will be two tools trying to achieve the same goal.)

mdtools.relstorage contains a relstorage variant of zodbupdate which claims to be much faster on relstorage as it can leverage parallelism.

Open issues

The pull request containing the migration strategy (zopefoundation/Zope#285) has to be extended for the other persistent classes in Zope. There have to be alike changes in all packages providing persistent classes.

Zope is welcome in the Python 3 wonderland!

Earl Zope already got the beta permission to stay in the Python 3 wonderland some months ago. His current objective is to help old friends to come to the Python 3 wonderland and to make new friends. He has to build trust in his will and ability to stay in the Python 3 wonderland.

The Zope-4-Welcome sprint last week was a great opportunity to work towards the final permission for Earl Zope. We were a group of 15 developers from different companies and backgrounds building applications on Zope in various ways.

We accomplished the following goals:

  • There are some old friends of Earl Zope. He thought that he no longer needs them in the Python 3 wonderland but other applications built on Zope need them, so they were pushed towards the new land:
  • Knight RestrictedPython got some love and a new beta release.
  • Earl Zope could help an old friend (a custom Zope 2.13 application) to get prepared to move to the new land.
  • Duchess CMFCore got a beta permission for the Python 3 wonderland including her beloved siblings:
  • Prince Plone is not yet ready to live in the Python 3 wonderland but he is already a welcome guest. It is only a matter of time before he will get an alpha permission:
    • The instance starts and many actions in the UI work pretty well.
    • The test story was brought some steps further so it is possible to start testing Plone under Python 3.
    • Details are described in a Blog post of Philip Bauer.
  • The migration of a toy Data.fs was tested and successfully completed. (Details will follow in another blog post.) The Migration took the following steps:
    • from Zope 2.13 on Python 2.7
    • via Zope 4 on Python 2.7
    • to Zope 4 on Python 3.6
  • The ZMI of Earl Zope got a facelift (Zope#249) which is not complete yet but looks promising.
  • And last but not least Earl Zope himself got the 5th extension of his beta permission: Zope 4.0b5.

Earl Zope says a hearty thank you to all who where involved in this sprint in Halle or remote by coding or providing the resources and time to code.

Welcome in the Python 3 wonderland!