Undo transactions by truncating ZODB Data.fs

truncate can be used to permanently set back a ZODB Data.fs to a certain point in transaction history.

Sometimes I break the Data.fs of my ZODB in a way that the Zope instance cannot start any more or I want to try again a migration. In such situations it is handy that writing to a Data.fs means extending the file at the end. So the unwanted transaction can be truncated. Normally I use the following steps to do so:

1. Install ZODB in a virtualenv

This is needed to get the script named fstail. If you are already using Python 3, call:
python3.7 -m venv v_zodb

If you are still on Python 2, call:
virtualenv-2.7 v_zodb

Caution: The Python major version (2 or 3) must match the version you are using for the Zope instance.

Install the script into the virtual environment using:
cd v_zodb
bin/pip install ZODB

2. Stop Zope and ZEO

Stop both Zope and (if used) the ZEO server. This is necessary for your cut to get noticed by client and server.

3. Find the position where to cut

Call fstail. With -n you are able to specify the number of transactions to be shown:
bin/fstail -n 20 path/to/Data.fs

fstail returns something like the following. Newer transaction are at the top: (These lines here are only some extracted from a longer output.)

2019-04-24 08:38:44.622984: hash=0b59c10e6eaa947b2ec0538e26d9b4f9128c03cb
user=' admin' description='/storage/58bdea07-666c-11e9-8a63-34363bceb816/edit' length=19180 offset=12296784 (+97)
2019-04-24 08:38:06.823673: hash=3a595fb50b913bad819f0d5bd8d152e06bc695d7
user=' admin' description='/portal/site/add-page-form' length=132830 offset=12121677 (+58)
2019-02-26 10:28:10.856626: hash=5b2b0fbc33b53875b7110f82b2fe1793245c590b
user=' admin' description='/index_html/pt_editAction' length=444 offset=11409587 (+54)

Using the provided information in user and description you should be able to find the transaction from which on newer transactions should be removed. You need the provided value after offset= to do the cut.

In my example above, if /portal/site/add-page-form is the faulty transaction, my cut point is 12121677.

4. 3, 2, 1 … cut

Caution: Every transaction after the cut point (including the one you took the offset from) will get removed by cutting.

Get a truncate tool of your choice. I am using here one of Folkert van Heusden which comes with MacPorts and claims to be command-line compatible with the (Free-)BSD version.

In my example I would call it using:
truncate -s 12121677 path/to/Data.fs

That’s all. Start ZEO and Zope again to be back in transaction history where you have cut.

References

Celebration: Zope 4 final release

TL;DR: Zope 4 beta phase ended, final version released!

After hard, long years of preparation Earl Zope now finally made it to get a permanent license for the Python 3 wonderland: In September 2016 almost 20 people started with the reanimation of Zope at the Zope Resurrection sprint. This marked the beginning of a wonderful journey for Earl Zope himself for the people who helped him. In August 2017 Earl Zope became aware that his Python 2 country will irreversibly be destroyed by 2020. Earl Zope was successfully applying for for a beta permission for the Python 3 wonderland in September 2017. This beta permission has been extended 9 times to give Earl Zope time to become a good citizen in his new home country.

Earl Zope says a big thank you to all who:

  • contributed to the Python 3 migration even before the resurrection sprint
  • wrote bug reports
  • fixed bugs
  • contributed time and/or money for the migration process
  • encouraged the developers
  • tested beta versions or even used them in production

To be welcome in the Python 3 wonderland many nuts had to be cracked:

  • porting of the code of Zope and its dependencies to Python 3
  • rewrite of RestrictedPython from scratch
  • develop a migration strategy for the ZODB contents aka Data.fs
  • polish the user interface of the Zope management interface (ZMI)
  • and many more…

Earl Zope is looking forward to a happy future in the Python 3 wonderland. Currently he did not yet give up his settling in the Python 2 land. This is planned to happen shortly before or after the Python 2 sunset in the beginning of 2020 when the son of Earl Zope IV becomes the new Earl Zope V. See the roadmap for details.

See the documentation how to install Zope. It also documents the migrations steps.

Zope Spring Cleaning: Last minute information

As the beta permission of Earl Zope in Python 3 wonderland was extended in October 2018, gocept invites Zope developers to the upcoming sprint from 08.05. till 10.05.2019 in Halle (Saale), Germany, to continue together on the work, which is still left.

We aim to polish the last dusty spots on Earl Zope for the final permission to Python 3 wonderland aka the final 4.0 release. As Plone and other applications based on Zope have finally found a way to migrate a ZODB Data.fs created with Python 2 to Python 3, the obstacles for this final permit are almost gone.

So if you have questions concerning migrating databases, it is a good time to join or open an issue on GitHub. As many people are working on Zope during these days, the probability of a quick answer is high.

As organizational tool to coordinate the work, we use GitHub projects again, as it allows cross-repository tracking of issues.

Our current schedule:

  • Wednesday
    • 8:15 Breakfast at gocept kitchen
    • 9:00 Welcome at gocept office and start sprinting afterwards
    • 12:30 Lunch
    • 13:30 Happy sprinting
    • between 15:00 and 16:00 coffee break
    • 18:00 Lights out
    • Going to a local pub
  • Thursday:
    • 8:15 Breakfast
    • 9:00 Standup
    • 12:30 Lunch
    • 13:30 Happy sprinting
    • between 15:00 and 16:00 coffee break
    • 17:00 A game of boules if the weather permits it
    • Going to a local pub
  • Friday:
    • 8:15 Breakfast
    • 9:00 Standup
    • 12:30 Lunch
    • 13:30 Happy sprinting
    • 15:00 Closing meeting
    • 16:00 Lights out

Parking: As Saltlabs in located in a pedestrian zone, the availability of parking spots is rather low. Please use one of the parking decks nearby.

One last hint: The location of the sprint is Leipziger Str. 70, Halle (Saale), Germany.

Zope spring cleaning sprint

The last year before the sunset of the Python 2 land has begun. Earl Zope feels relatively well prepared to live in the Python 3 wonderland.

Some issue are still open which are required for the final permission to stay in Python 3 wonderland:

  • test and improve ZODB migration
  • update the documentation
  • fix some glitches in the visual appearance of Earl Zope
  • fix some issues occurring while working together with other residents of the Python 3 wonderland

We invite you, to join forces for three days in May at Saltlabs in Halle and keep on developing good (database) migration stories for various Zope projects, you might have. So bring your Data.fs and let’s port it to Python.

Sprint information

  • Date: Wednesday, 8th until Friday, 10th of May 2019
  • Location: Leipziger Str. 70, Halle (Saale), Germany

Registration

Please join us via Meetup.

Saltlabs Sprint: last minute information

Earl Zope is now nearly settled down in Python 3 wonderland. On the Zope and Plone sprint from Monday, 1st until Friday, 5th of October 2018 in Halle (Saale), Germany we will work towards the final Zope 4 release aka the final permission for the Python 3 wonderland.

We are currently 33 participants for the sprint. So be prepared for a huge sprint with many interesting people. The Saltlabs have a café (called KOFFIJ) we can use, a big meeting room with big display (aka the Thronsaal) and many smaller rooms including the offices of gocept. So there will be enough room to work in bigger and smaller groups.

To keep the organisational overhead low with this amount of participants, we plan to separate in two teams: Zope and Plone. Those teams will organise themselves individually and we will have a short daily meeting after lunch to share the status in a condensed manner with the other team. Direct communication in case of a difficult problem is, of course, always possible.

We reserved up to one hour after the daily meeting for talks and presentations by you about interesting topics around Zope and Plone, successful migration stories, or something else you want to share with the community. So if you have some interesting slides, please bring them with you and register during the week for a slot.

Our current schedule:

  • Sunday
    • 19:00, there is a table reserved at Grober Gottlieb, so if you’ve already arrived and want some company, you are invited to join.
  • Monday
    • 9:00 Breakfast at KOFFIJ (This is the café in the ground floor of Saltlabs aka the window to the left on the picture above.)
    • 10:00 Welcome at KOFFIJ and start sprinting afterwards
    • 12:30 Lunch
    • 13:30 Sprint planning and introduction for all sprinters at Thronsaal
    • between 15:00 and 17:00 coffee break at KOFFIJ
    • 18:00 Lights out
  • All other days:
    • 8:30 Breakfast
    • 9:00 Standup in the team (Zope, Plone)
    • 12:30 Lunch
    • 13:30 Daily meeting at Thronsaal
    • 14:00 (Lightning) Talks at Thronsaal
    • between 15:00 and 17:00 coffee break at KOFFIJ
    • 18:00 Lights out
  • Tuesday:
    • 11:00 till 17:00 Massages, there will be a list to register on Monday
    • 19:00 social evening at Eigenbaukombinat (local hacker space) with pizza, beer and mate
  • Friday:
    • 13:30 Closing meeting with presentations at Thronsaal
    • 17:00 Lights out

If you cannot make it to the Welcome meeting, ask at KOFFIJ for one of the gocept staff to get a personal introduction.

Parking: As Saltlabs in located in a pedestrian zone, the availability of parking spots is rather low. Please use one of the parking decks nearby.

As organizational tool to coordinate the work, we try to use Github projects this time, as it allows cross-repository tracking of issues.

One last hint: The location of the sprint is Leipziger Str. 70, Halle (Saale), Germany.

Saltlabs Sprint: Zope and Plone sprint in a new location

After Earl Zope II is now nearly relocated to the Python 3 wonderland, gocept will move to a new head quarter in the next months. This is the right time to celebrate with a new sprint, as we have now even more space for sprinters. The new location is called the “Saltlabs”, a place for IT companies in Halle (Saale), Germany.

Sprint information

  • Date: Monday, 1st until Friday, 5th of October 2018
  • Location: Leipziger Str. 70, Halle (Saale), Germany

Sprint topics

This sprint has three main topics:

Create a final Zope 4 release

Before releasing a final version of Zope 4 we want to resolve about at least 40 issues: Some bugs have to be fixed,  some functions have to be polished and documentation has to be written resp. reviewed. On the other hand there is the re-brush of the ZMI using Bootstrap which should be completed beforehand, as it modernizes the ZMI and allows for easier customisation, but might also be backwards incompatible with certain test suites. There is an Etherpad to write down ideas, tasks, wishes and work proposals, which are not currently covered by the issue tracker.

Port Plone to Python 3

The following tasks are currently open and can be fixed at the sprint:

  • successfully run all Plone tests and even the robotframework tests on Python 3
  • Zope 4 lost the WebDAV support: find resp. create a replacement
  • document the WSGI setup and test it in a production ready environment
  • port as many as possible add-ons to Python 3 (e.g. Mosaic and Easyform)
  • work on the Migration of ZODB contents (Data.fs) to Python 3
  • improve the test setup with tox.
  • start to support Python 3.7

Polish Plone 5.2

The upcoming Plone 5.2 release will appreciate some love and care at the following items:

  • new navigation with dropdown and better performance
  • Barceloneta theme: ease the customisation and improve responsiveness
  • parallelise the tests so they run faster
  • remove Archetypes and other obsolete packages

See also the list of topics on plone.org for this sprint.

Organisational Remarks

In order to coordinate the participation for this sprint, we ask you to join us on Meetup. We can then coordinate the catering and requirements for space.

As this sprint will be running longer than usual (five days), it is also possible to join only for a part of the week. As October 3rd is the national holiday, we are trying to organise some social event for those who are interested in having a small break.

For a better overview, please indicate your participation also on this doodle poll.

 

Migrate a Zope ZODB Data.fs to Python 3

TL;DR Use zodbupdate.

Problem

A ZODB Data.fs which was created under Python 2 cannot be opened under Python 3. This is prevented by using a different magic code in the first bytes of the file. This is done on purpose because str has a different meaning for the two Python versions: Under Python 2 a str is a container for characters with an arbitrary encoding (aka bytes​). Python 3 knows str as a text datatype which was called unicode in Python 2. Trying to load a str object in Python 3 which actually contains binary data will fail. It has to be bytes, but bytes is an alias for str  in Python 2 which means Python 2 replaces bytes  with str making is impossible to give Python 3 the class it expects for binary data. A Python 2 str  with an arbitrary encoding will break, too.

Solution

The Data.fs has to be migrated: each str  which actually contains bytes has to be converted into a zodbpickle.binary object which deserialises as bytes under Python 3. The str objects actually containing text have to be decoded to unicode. There are currently two tools which claim that they are able to do such a migration:

  • zodb.py3migrate was already written at Berlin Strategic sprint in 2016, but it was never able to prove that it can do what it claims: At the time when it was written there was no Zope which could run on Python 3. Now as we have Zope 4 running on Python 3 it does not seem to do its conversion job quite well: I was able to migrate a toy database but had to catch an unpickling error.
  • zodbupdate was enriched by a Python 3 migration. A big thank you to Sylvain Viollon and the developers at Minddistrict! It has proven its claims! At the Zope 4 welcome sprint I was able to migrate a Data.fs created on Zope 2.13 running on Python 2 to Zope 4 running on Python 3.

Steps

  1. Migrate your Zope application to Zope 4. (zodbupdate  requires at least ZODB 4 which is not the default ZODB version of Zope 2.13) — For my toy database containing only a file object and an image this was no problem. Zope 4  is starting with such a database. It might show some broken objects because Zope no longer depends on some previous core packages like Products.Sessions. If your application needs those packages you should add them to your Zope environment.
  2. ​zodbupdate has to be installed into the Zope 4 environment so it can access the Python classes. (It has to read the pickles in the ZODB.)
  3. There needs to be an entry_point in setup.py for each package which contains persistent Python classes. The entry point has to be named "zodbupdate.decode" and needs to point to a dictionary mapping paths to  str attributes to a conversion (bytes resp. a specific encoding). For Details see the migration documentation of zodbupdate. I prepared a branch of Zope 4 which contains this configuration dictionary for OFS.Image and OFS.File, see zopefoundation/Zope#285.
  4. Run zodbupdate --pack --convert-py3 on the Data.fs using Python 2.
  5. Copy the Data.fs over to the Zope 4 instance running on Python 3. Data.fs.index will be discarded at the first start. (There is an error message telling that it cannot be read.)
  6. Enjoy the contents of the Data.fs running on Python 3.

Conclusion

It is possible (proven for a toy database) to migrate a Data.fs from Zope 2.13  (Python 2) to Zope 4 (Python 3).

zodbupdate is the way to go. Although it cannot do the migration completely autonomously the developers of Python packages can provide migration configuration in their packages which can be used in the migration step so the configuration has only to be written once.

zodb.py3migrate has an analysis step which shows the attribute names where the str objects are stored. (This could be added to zodbupdate, so do not expect that there will be two tools trying to achieve the same goal.)

mdtools.relstorage contains a relstorage variant of zodbupdate which claims to be much faster on relstorage as it can leverage parallelism.

Open issues

The pull request containing the migration strategy (zopefoundation/Zope#285) has to be extended for the other persistent classes in Zope. There have to be alike changes in all packages providing persistent classes.