Saltlabs Sprint: Zope and Plone sprint in a new location

After Earl Zope II is now nearly relocated to the Python 3 wonderland, gocept will move to a new head quarter in the next months. This is the right time to celebrate with a new sprint, as we have now even more space for sprinters. The new location is called the “Saltlabs”, a place for IT companies in Halle (Saale), Germany.

Sprint information

  • Date: Monday, 1st until Friday, 5th of October 2018
  • Location: Leipziger Str. 70, Halle (Saale), Germany

Sprint topics

This sprint has three main topics:

Create a final Zope 4 release

Before releasing a final version of Zope 4 we want to resolve about at least 40 issues: Some bugs have to be fixed,  some functions have to be polished and documentation has to be written resp. reviewed. On the other hand there is the re-brush of the ZMI using Bootstrap which should be completed beforehand, as it modernizes the ZMI and allows for easier customisation, but might also be backwards incompatible with certain test suites. There is an Etherpad to write down ideas, tasks, wishes and work proposals, which are not currently covered by the issue tracker.

Port Plone to Python 3

The following tasks are currently open and can be fixed at the sprint:

  • successfully run all Plone tests and even the robotframework tests on Python 3
  • Zope 4 lost the WebDAV support: find resp. create a replacement
  • document the WSGI setup and test it in a production ready environment
  • port as many as possible add-ons to Python 3 (e.g. Mosaic and Easyform)
  • work on the Migration of ZODB contents (Data.fs) to Python 3
  • improve the test setup with tox.
  • start to support Python 3.7

Polish Plone 5.2

The upcoming Plone 5.2 release will appreciate some love and care at the following items:

  • new navigation with dropdown and better performance
  • Barceloneta theme: ease the customisation and improve responsiveness
  • parallelise the tests so they run faster
  • remove Archetypes and other obsolete packages

Organisational Remarks

In order to coordinate the participation for this sprint, we ask you to join us on Meetup. We can then coordinate the catering and requirements for space.

As this sprint will be running longer than usual (five days), it is also possible to join only for a part of the week. As October 3rd is the national holiday, we are trying to organise some social event for those who are interested in having a small break.

For a better overview, please indicate your participation also on this doodle poll.

 

Migrate a Zope ZODB Data.fs to Python 3

TL;DR Use zodbupdate.

Problem

A ZODB Data.fs which was created under Python 2 cannot be opened under Python 3. This is prevented by using a different magic code in the first bytes of the file. This is done on purpose because str has a different meaning for the two Python versions: Under Python 2 a str is a container for characters with an arbitrary encoding (aka bytes​). Python 3 knows str as a text datatype which was called unicode in Python 2. Trying to load a str object in Python 3 which actually contains binary data will fail. It has to be bytes, but bytes is an alias for str  in Python 2 which means Python 2 replaces bytes  with str making is impossible to give Python 3 the class it expects for binary data. A Python 2 str  with an arbitrary encoding will break, too.

Solution

The Data.fs has to be migrated: each str  which actually contains bytes has to be converted into a zodbpickle.binary object which deserialises as bytes under Python 3. The str objects actually containing text have to be decoded to unicode. There are currently two tools which claim that they are able to do such a migration:

  • zodb.py3migrate was already written at Berlin Strategic sprint in 2016, but it was never able to prove that it can do what it claims: At the time when it was written there was no Zope which could run on Python 3. Now as we have Zope 4 running on Python 3 it does not seem to do its conversion job quite well: I was able to migrate a toy database but had to catch an unpickling error.
  • zodbupdate was enriched by a Python 3 migration. A big thank you to Sylvain Viollon and the developers at Minddistrict! It has proven its claims! At the Zope 4 welcome sprint I was able to migrate a Data.fs created on Zope 2.13 running on Python 2 to Zope 4 running on Python 3.

Steps

  1. Migrate your Zope application to Zope 4. (zodbupdate  requires at least ZODB 4 which is not the default ZODB version of Zope 2.13) — For my toy database containing only a file object and an image this was no problem. Zope 4  is starting with such a database. It might show some broken objects because Zope no longer depends on some previous core packages like Products.Sessions. If your application needs those packages you should add them to your Zope environment.
  2. ​zodbupdate has to be installed into the Zope 4 environment so it can access the Python classes. (It has to read the pickles in the ZODB.)
  3. There needs to be an entry_point in setup.py for each package which contains persistent Python classes. The entry point has to be named "zodbupdate.decode" and needs to point to a dictionary mapping paths to  str attributes to a conversion (bytes resp. a specific encoding). For Details see the migration documentation of zodbupdate. I prepared a branch of Zope 4 which contains this configuration dictionary for OFS.Image and OFS.File, see zopefoundation/Zope#285.
  4. Run zodbupdate --pack --convert-py3 on the Data.fs using Python 2.
  5. Copy the Data.fs over to the Zope 4 instance running on Python 3. Data.fs.index will be discarded at the first start. (There is an error message telling that it cannot be read.)
  6. Enjoy the contents of the Data.fs running on Python 3.

Conclusion

It is possible (proven for a toy database) to migrate a Data.fs from Zope 2.13  (Python 2) to Zope 4 (Python 3).

zodbupdate is the way to go. Although it cannot do the migration completely autonomously the developers of Python packages can provide migration configuration in their packages which can be used in the migration step so the configuration has only to be written once.

zodb.py3migrate has an analysis step which shows the attribute names where the str objects are stored. (This could be added to zodbupdate, so do not expect that there will be two tools trying to achieve the same goal.)

mdtools.relstorage contains a relstorage variant of zodbupdate which claims to be much faster on relstorage as it can leverage parallelism.

Open issues

The pull request containing the migration strategy (zopefoundation/Zope#285) has to be extended for the other persistent classes in Zope. There have to be alike changes in all packages providing persistent classes.

Zope is welcome in the Python 3 wonderland!

Earl Zope already got the beta permission to stay in the Python 3 wonderland some months ago. His current objective is to help old friends to come to the Python 3 wonderland and to make new friends. He has to build trust in his will and ability to stay in the Python 3 wonderland.

The Zope-4-Welcome sprint last week was a great opportunity to work towards the final permission for Earl Zope. We were a group of 15 developers from different companies and backgrounds building applications on Zope in various ways.

We accomplished the following goals:

  • There are some old friends of Earl Zope. He thought that he no longer needs them in the Python 3 wonderland but other applications built on Zope need them, so they were pushed towards the new land:
  • Knight RestrictedPython got some love and a new beta release.
  • Earl Zope could help an old friend (a custom Zope 2.13 application) to get prepared to move to the new land.
  • Duchess CMFCore got a beta permission for the Python 3 wonderland including her beloved siblings:
  • Prince Plone is not yet ready to live in the Python 3 wonderland but he is already a welcome guest. It is only a matter of time before he will get an alpha permission:
    • The instance starts and many actions in the UI work pretty well.
    • The test story was brought some steps further so it is possible to start testing Plone under Python 3.
    • Details are described in a Blog post of Philip Bauer.
  • The migration of a toy Data.fs was tested and successfully completed. (Details will follow in another blog post.) The Migration took the following steps:
    • from Zope 2.13 on Python 2.7
    • via Zope 4 on Python 2.7
    • to Zope 4 on Python 3.6
  • The ZMI of Earl Zope got a facelift (Zope#249) which is not complete yet but looks promising.
  • And last but not least Earl Zope himself got the 5th extension of his beta permission: Zope 4.0b5.

Earl Zope says a hearty thank you to all who where involved in this sprint in Halle or remote by coding or providing the resources and time to code.

Welcome in the Python 3 wonderland!

“allow-hosts” in buildout considered harmful

Today we had the following error message when re-installing a project from scratch:

 While:
   Installing.
   Getting section application.
   Initializing section application.
   Installing recipe zc.zope3recipes.
   Getting distribution for 'zc.zope3recipes==0.13.0'.
 Error: Couldn't find a distribution for 'zc.zope3recipes==0.13.0'.

Yes this is a really old recipe but it still exists on PyPI. We are using zc.buildout in Version 2.10, and do not use a custom index. So being forced to use HTTPS to access PyPI does not seem be the problem.

After searching way too long we found that .buildout/default.cfg contains the following statement:

allow-hosts =
   *.python.org
   *.gocept.com
   *.gocept.net
   effbot.org
   dist.plone.org

It restricts the allowed hosts for download but it seems to restrict the index, too. https://pypi.python.org/simple nowadays redirects to https://pypi.org/simple which is not on the list.

Suggestion: Remove allow-hosts if possible. It is more harmful than good, especially because packages are nowadays downloaded from https://files.pythonhosted.org.

UPDATE:

I filed an issue for zc.buildout at GitHub: buildout/buildout#447

A heartily welcome for Zope in the Python 3 wonderland

Once upon the time there was Earl Zope II. A wise guy was telling him that his world will come to an end. He found out that this was true that he had only some years to prepare to immigrate to the Python 3 wonderland.

His preparation was successful: He got past the strict immigration check and has now a beta permission to stay in the Python 3 wonderland. Earl Zope really likes his new home, but he is missing some friends. Most of them are still at the border of Python 3 wonderland and have to go through the immigration process. Earl Zope would be pleased to offer the same service to his friends as in the old Python 2 land.

To get the final permission to stay in Python 3 wonderland Earl Zope needs to build up trust in his abilities and his stability by other inhabitants and old friends. Therefore, he has to show, that he can work with old and new friends in the Python 3 wonderland.

Zope 4 Welcome sprint – 16th until 18th May 2018

Earl Zope invites you to join a sprint with some helpful people to welcome Zope 4 and his friends in Python 3 wonderland. This means: Bring in your Zope 2 based application and we look together how to port it to Zope 4 or even Python 3. That’s why we call the sprint “Zope 4 Welcome sprint”. You can also help by posting issues or even pull requests about your migration attempts.

Additionally we look forward to work on an improved version of the ZMI (Zope Management Interface) and to fix some bugs preventing Earl Zope from getting the final permission.

The sprint will be located in Halle (Saale), Germany. We meet there from Wednesday, 16th till Friday, 18th of May 2018. Please join us via Meetup even if you are planning to work from remote. You will find more detailed and updated information about the sprint there, too.

Zope – undo transactions via debug console

If you cannot access the root level in Zope 2 via the browser but you are able to use the debug console you have enough to undo transactions.

Start the debug console:

$ bin/zinstance debug

List the transaction descriptions, user names and ids of the last 10 transactions:

>>> from pprint import pprint
>>> pprint(app._p_jar.db().undoInfo(0, 10))

(0 is the youngest transaction, the first one to show)

Undo some transactions:

>>> import transaction
>>> transaction.get().note(u"Undo transactions: ...")
>>> app._p_jar.db().undoMultiple([id1, id2])
>>> transaction.commit()

(id1, id2 have to be the actual id strings from the transaction listing.)

Be aware:

  • Use a sorted list of transaction ids, start with the id of the youngest transaction you want to undo.
  • You cannot undo transactions those objects were changed again later without undoing those transactions, too.
  • The actual undo is executed when calling transaction.commit(). (This might take some time to execute.)
  • You can check the result of your undo using the transaction log as undo only creates an inverse transaction.

 

This blog post first appeared on icemac15.wordpress.com.

Migrating from Zope 2.13 to 4.0b2

Beta-Testing Zope 4 together with PerFact Innovation

TL;DR: There are some rough edges when migrating an existing Data.fs and needed Python code from Zope 2.13 to Zope 4 but there is nothing what cannot be solved.

The story

The German company PerFact Innovation (www.perfact-innovation.de) has a customer product built on Zope 2.13. The code needed for the customisation to different customers is stored in the ZODB. They invited me to a workshop where we looked into the migration story to Zope 4. We used the just released Zope 4.0b2.

After half a day the core functions of the customer product where working including the switch from ZServer to WSGI. There were some rough edges we had to come around. I list them here as a reference for other people who start migrating their code to Zope 4.

We also poke around with a fresh Zope 4 installation on Python 3 to see what this will bring. It was nice to see that a DTMLMethod calling another one while mixing bytes and unicode  no longer leads to strange encoding problems but keeps bytes  as bytes.

Expect most of the bugs and some of the uglinesses listed below to be fixed in Zope 4.0b3.

Bugs

There are some bugs we could easily hack around in the workshop. Most of them are even fixed by now:

  • Some objects cannot be loaded from the ZODB because their classes used to be old-style classes but in Zope 4.0b2 they became new-style classes. Fixed in [zopefoundation/Zope#205]
  • Python scripts cannot be called because the compiled code is expected on a different attribute. This requires re-compiling them. Fixed in [zopefoundation/Products.PythonScripts#12]
  • Some modules seem no longer allowed to be used in through the web PageTemplates, e. g. Products.PythonScripts.standard. Fixed in [zopefoundation/Zope#209]
  • Accessing METAL macros using the getitem style (e. g. metal:use-macro=python:here.master.macros[‘foo’]“) is not allowed. Its Python class seems to be missing a security declaration. Reported in [zopefoundation/Zope#210]
  • When using Python 3 it is not possible to create a new Script (Python) in the ZMI because the default script code is not Python 3 compatible. Already fixed in [zopefoundation/Products.PythonScripts#10] awaiting release.

Uglinesses

Some things are still a bit raw in Zope 4 but for a beta version it is possible to live with them:

Breaking changes

We were facing some of the breaking changes in Zope 4, which will require further development if the customer product should behave and be developed like before:

  • Support for WebDAV, XML-RPC and FTP has been moved to the ZServer package. This means a WSGI based installation does not provide support for these protocols. Maybe WSGI middlewares could be written to support them again individually.
  • The products directive is no longer supported in the zope.conf (only via ZServer). This means that the contents of the Products directory inside the Zope instance directory have to become Python packages which can be installed via pip resp. zc.buildout.

Required changes

We were facing some required changes on different levels to get the code running:

Conclusion

It is not impossible to migrate an existing Data.fs to Zope 4 (while staying on Python 2). It is even easier if the through the web approach was used to create the Data.fs as only very little code has to be modified that way.

Please test your applications on Zope 4 during the beta phase (until fall 2018) so we will get a final release which will be ready for production usage. If you have questions we are here to help.