After Earl Zope II is now nearly relocated to the Python 3 wonderland, gocept will move to a new head quarter in the next months. This is the right time to celebrate with a new sprint, as we have now even more space for sprinters. The new location is called the “Saltlabs”, a place for IT companies in Halle (Saale), Germany.
- Date: Monday, 1st until Friday, 5th of October 2018
- Location: Leipziger Str. 70, Halle (Saale), Germany
This sprint has three main topics:
Create a final Zope 4 release
Before releasing a final version of Zope 4 we want to resolve about at least 40 issues: Some bugs have to be fixed, some functions have to be polished and documentation has to be written resp. reviewed. On the other hand there is the re-brush of the ZMI using Bootstrap which should be completed beforehand, as it modernizes the ZMI and allows for easier customisation, but might also be backwards incompatible with certain test suites. There is an Etherpad to write down ideas, tasks, wishes and work proposals, which are not currently covered by the issue tracker.
Port Plone to Python 3
The following tasks are currently open and can be fixed at the sprint:
- successfully run all Plone tests and even the robotframework tests on Python 3
- Zope 4 lost the WebDAV support: find resp. create a replacement
- document the WSGI setup and test it in a production ready environment
- port as many as possible add-ons to Python 3 (e.g. Mosaic and Easyform)
- work on the Migration of ZODB contents (
Data.fs) to Python 3
- improve the test setup with
- start to support Python 3.7
Polish Plone 5.2
The upcoming Plone 5.2 release will appreciate some love and care at the following items:
- new navigation with dropdown and better performance
- Barceloneta theme: ease the customisation and improve responsiveness
- parallelise the tests so they run faster
- remove Archetypes and other obsolete packages
See also the list of topics on plone.org for this sprint.
In order to coordinate the participation for this sprint, we ask you to join us on Meetup. We can then coordinate the catering and requirements for space.
As this sprint will be running longer than usual (five days), it is also possible to join only for a part of the week. As October 3rd is the national holiday, we are trying to organise some social event for those who are interested in having a small break.
For a better overview, please indicate your participation also on this doodle poll.
TL;DR Use zodbupdate.
Data.fs which was created under Python 2 cannot be opened under Python 3. This is prevented by using a different magic code in the first bytes of the file. This is done on purpose because
str has a different meaning for the two Python versions: Under Python 2 a
str is a container for characters with an arbitrary encoding (aka
bytes). Python 3 knows
str as a text datatype which was called
unicode in Python 2. Trying to load a
str object in Python 3 which actually contains binary data will fail. It has to be
bytes is an alias for
str in Python 2 which means Python 2 replaces
str making is impossible to give Python 3 the class it expects for binary data. A Python 2
str with an arbitrary encoding will break, too.
Data.fs has to be migrated: each
str which actually contains
bytes has to be converted into a
zodbpickle.binary object which deserialises as
bytes under Python 3. The
str objects actually containing text have to be decoded to
unicode. There are currently two tools which claim that they are able to do such a migration:
- zodb.py3migrate was already written at Berlin Strategic sprint in 2016, but it was never able to prove that it can do what it claims: At the time when it was written there was no Zope which could run on Python 3. Now as we have Zope 4 running on Python 3 it does not seem to do its conversion job quite well: I was able to migrate a toy database but had to catch an unpickling error.
- zodbupdate was enriched by a Python 3 migration. A big thank you to Sylvain Viollon and the developers at Minddistrict! It has proven its claims! At the Zope 4 welcome sprint I was able to migrate a
Data.fs created on
Zope 2.13 running on Python 2 to
Zope 4 running on Python 3.
- Migrate your Zope application to
Zope 4. (
zodbupdate requires at least
ZODB 4 which is not the default ZODB version of
Zope 2.13) — For my toy database containing only a file object and an image this was no problem.
Zope 4 is starting with such a database. It might show some broken objects because Zope no longer depends on some previous core packages like
Products.Sessions. If your application needs those packages you should add them to your Zope environment.
zodbupdate has to be installed into the
Zope 4 environment so it can access the Python classes. (It has to read the pickles in the ZODB.)
- There needs to be an
setup.py for each package which contains persistent Python classes. The entry point has to be named
"zodbupdate.decode" and needs to point to a dictionary mapping paths to
str attributes to a conversion (
bytes resp. a specific encoding). For Details see the migration documentation of zodbupdate. I prepared a branch of
Zope 4 which contains this configuration dictionary for
OFS.File, see zopefoundation/Zope#285.
zodbupdate --pack --convert-py3 on the
Data.fs using Python 2.
- Copy the
Data.fs over to the
Zope 4 instance running on Python 3.
Data.fs.index will be discarded at the first start. (There is an error message telling that it cannot be read.)
- Enjoy the contents of the
Data.fs running on Python 3.
It is possible (proven for a toy database) to migrate a
Zope 2.13 (Python 2) to
Zope 4 (Python 3).
zodbupdate is the way to go. Although it cannot do the migration completely autonomously the developers of Python packages can provide migration configuration in their packages which can be used in the migration step so the configuration has only to be written once.
zodb.py3migrate has an analysis step which shows the attribute names where the
str objects are stored. (This could be added to
zodbupdate, so do not expect that there will be two tools trying to achieve the same goal.)
mdtools.relstorage contains a
relstorage variant of
zodbupdate which claims to be much faster on
relstorage as it can leverage parallelism.
The pull request containing the migration strategy (zopefoundation/Zope#285) has to be extended for the other persistent classes in Zope. There have to be alike changes in all packages providing persistent classes.
Beta-Testing Zope 4 together with PerFact Innovation
TL;DR: There are some rough edges when migrating an existing Data.fs and needed Python code from Zope 2.13 to Zope 4 but there is nothing what cannot be solved.
The German company PerFact Innovation (www.perfact-innovation.de) has a customer product built on Zope 2.13. The code needed for the customisation to different customers is stored in the ZODB. They invited me to a workshop where we looked into the migration story to Zope 4. We used the just released Zope 4.0b2.
After half a day the core functions of the customer product where working including the switch from ZServer to WSGI. There were some rough edges we had to come around. I list them here as a reference for other people who start migrating their code to Zope 4.
We also poke around with a fresh Zope 4 installation on Python 3 to see what this will bring. It was nice to see that a
DTMLMethod calling another one while mixing
unicode no longer leads to strange encoding problems but keeps
Expect most of the bugs and some of the uglinesses listed below to be fixed in Zope 4.0b3.
There are some bugs we could easily hack around in the workshop. Most of them are even fixed by now:
- Some objects cannot be loaded from the ZODB because their classes used to be old-style classes but in Zope 4.0b2 they became new-style classes. Fixed in [zopefoundation/Zope#205]
- Python scripts cannot be called because the compiled code is expected on a different attribute. This requires re-compiling them. Fixed in [zopefoundation/Products.PythonScripts#12]
- Some modules seem no longer allowed to be used in through the web PageTemplates, e. g. Products.PythonScripts.standard. Fixed in [zopefoundation/Zope#209]
- Accessing METAL macros using the
getitem style (e. g. metal:use-macro=“python:here.master.macros[‘foo’]“) is not allowed. Its Python class seems to be missing a security declaration. Reported in [zopefoundation/Zope#210]
- When using Python 3 it is not possible to create a new
Script (Python) in the ZMI because the default script code is not Python 3 compatible. Already fixed in [zopefoundation/Products.PythonScripts#10] awaiting release.
Some things are still a bit raw in Zope 4 but for a beta version it is possible to live with them:
We were facing some of the breaking changes in Zope 4, which will require further development if the customer product should behave and be developed like before:
- Support for WebDAV, XML-RPC and FTP has been moved to the ZServer package. This means a WSGI based installation does not provide support for these protocols. Maybe WSGI middlewares could be written to support them again individually.
products directive is no longer supported in the
zope.conf (only via ZServer). This means that the contents of the
Products directory inside the Zope instance directory have to become Python packages which can be installed via
We were facing some required changes on different levels to get the code running:
- Installation: Some Python packages are no longer dependencies of Zope or they have been extracted to separate packages and Zope does not depend on them. They have to be installed separately when they are needed. These packages are:
- Products code: Zope 4 no longer contains the
Globals module. In Zope 2.13 it only contained imports to keep the backwards compatibility with older Zope releases.
Globals is gone now. To find out how to change your imports look at the Globals module on the 2.13 branch.
- PageTemplates: PageTemplates are now rendered using Chameleon. This prevents you from using
-- in HTML comments as it is not allowed there. (See also the HTML comments document of the WebDesignGroup.)
- Error handling:
standard_error_message does not exist any more in Zope 4. It only supports exception views which have to be written in file system code. See Catching and rendering exceptions as a tutorial how to create exception views and even a hack to restore using
It is not impossible to migrate an existing Data.fs to Zope 4 (while staying on Python 2). It is even easier if the through the web approach was used to create the Data.fs as only very little code has to be modified that way.
Please test your applications on Zope 4 during the beta phase (until fall 2018) so we will get a final release which will be ready for production usage. If you have questions we are here to help.
Once upon the time there was Earl Zope II. His lands where threatened by a huge disaster called “Python 2 sunset”. His only chance was to emigrate to the Python 3 wonderland. After a long preparation phase for himself and his courtiers he was able to move to the new land. But the Python 3 wonderland has strict immigration authorities: They only allow “compatible” fellows to get a residency permit. The permission itself has three levels:
- alpha – To get this level the candidate has to prove that he can breath the Python 3 air and drink the water in this land without getting falling over. Earl Zope II needed a while to acclimate in the Python 3 wonderland as its climate is a bit different from what he was used to.
- beta – This level requires the the acclimation phase was successful and people could start to rely that the candidate will stay in the new land. Earl Zope II reached this level by proving that all his courtiers – he relies on – are ready for this level.
- final – To reach this level the candidate needs to prove that he is living in successful relations in Python 3 wonderland. Other inhabitants must be able to trust him and the services he offers. After Earl Zope II now has reached the beta phase he is now able to offer his services and hope that he gets the final level about fall 2018.
Back in his own country Earl Zope II had the official name “Zope2 2.13”. Before the beta phase of his immigration he thought that he would have to change the name to something like “Zope2 4.0b1”. This is looks ugly and some people protested against this name. With the kind and quick help of Baiju Muthukadan Earl Zope II was able to change his official name to “Zope 4.0b1”. Thank you very much Baiju Muthukadan! Earl Zope is very proud of this new shorter name and is heartily thankful for this opportunity.
The release of the first Beta version means, that no currently existing features will be removed until the final version. But There will be some new features and many bugfixes.
New Features of Zope version 4.0
- Support for Python 3.4 up to 3.6: Currently Python 3 can only be used for new projects. There is only an experimental way to convert an existing ZODB from Python 2 to 3 as it is not possible to run the same
Data.fs under both versions, see zodb.py3migrate.
- Zope now by default runs as a WSGI application. The previously used
ZServer is still supported but only runs under Python 2.
- Chameleon based templates are now the default.
- The name of the distribution changed from
Zope. The previous
Zope2 package will remain as a meta package which depends on
Zope. This allows packages which require Zope 4.x to depend on
Zope instead of
- Removed deprecated code and BBB imports like the
Globals package or the internal help system.
- Some smaller features, bug fixes and security fixes, see the Change log.
A big “Thank you!” to all who made this release possible:
- dedicated people investing time, thoughts and money
- nice companies allowing their people to participate on Zope sprints
- the Plone Foundation sponsoring Zope sprints
We had a great Zope 4 Phoenix Sprint helping to raise Zope from the ashes! Thank you to everybody who participated in Halle or from remote.
Beta one of Zope is out:
We need the feedback to adapt Zope to the needs in the wild. The current plan is to create new beta releases once a while after implementing features resp. bug fixes or on demand. The current plan is to release a final 4.0 version in fall of 2018. This should allow software projects built upon Zope to migrate there code before the Python 2 sunset in 2020.
Today we migrated the documentation of zodb.py3migrate from pythonhosted.org to zodbpy3migrate.readthedocs.io.
This requires a directory – for this example I name it redir – containing a file named index.html with the following content:
content="0; url=http://zodbpy3migrate.rtfd.io" />
Redirect to zodbpy3migrate.rtfd.io
To upload it to pythonhosted.org I called:
py27 setup.py upload_docs --upload-dir=redir
Now pythonhosted.org/zodb.py3migrate points to read the docs.
Credits: The HTML was taken from the Trello board of the avocado-framework.
UPDATE: The approach described here no longer works as the required API has been shut down. See pypa/warehouse#582.