Tag: programming

The final teardown

After writing so many teardowns in tests for software projects for customers, open source projects and in house software, it is now time for the gocept folks to finally work on a last teardown of the company itself. After 23 years of existence gocept will close the doors and stop operating business at 2023-11-30. The developers found a new home at the wonderful company Minddistrict. There we will work in a product team and build software to empower people to take care of their mental well-being.

Customers

We want to thank all our customers for their trust in us over the years and for the often multi year long collaboration. Sometimes gocept was a bit on the bleeding edge of technology and often that was shining through in the development work. But we did our best to provide the individual software applications to the needs of our customers.

Open Source Software

It was in the very essence of gocept to work with open source software. The company grew with Zope and contributed to the Zope ecosystem. Out of this relationship many events were supported like DZUG-Tagung, a EuroPython Conference and in particular in the last years a round of Zope sprint to ensure compatibility and stability in the Python 3 wonderland. We are very happy, that Minddistrict also builds its core product on zope.* packages and that we can further contribute to the open source world.

Most of the gocept.* packages on https://github.com/gocept have been archived or given to the people and companies, that still use them. In case you need to work with them, please contact mail@gocept.com or the respective maintainers on PyPI. We will be able to unarchive the package and if necessary transfer the release rights.

gocept folks

Some parts of gocept already left the company in 2014 and formed https://flyingcircus.io/. They are more successful than ever providing a very good and individual hosting and, more important, operation solutions for bigger applications and complex setups. They took over the dedication to software and to new technology, and they are good colleagues to work with.

The remaining 4 developers at gocept join the Minddistrict company and will work there in a bigger team to empower people to take care of their mental well-being in the Netherlands, in Germany and the DACH-region. We will still keep an office in Halle, so there might be a chance to stop by.

The last Christian of gocept (in a row of many) will continue his management work within the Saltlabs and the Koffij – with the mission to develop modern, contemporary and flexible workplaces in Halle and build communities that motivate and offer pleasant working environments.

gocept.cleanup()

In case you need anything or want to reach us, please find us on GitHub (or the place to be in the future) or write an email to mail@gocept.com.

Ready, set, sprint: Earl Zope invites again

Sprint with Earl Zope at September, 21st and 22nd of 2023.

Earl Zope is already a long time inhabitant of the Python 3 wonderland. Living here for some years with the help of his friends, he sometimes looks back to the “good old times” but mostly enjoys his current live with all its surprises and unexpected turns.

Gocept invites you to sprint for Earl Zope’s future well-being: 21st till 22nd of September 2023 in the SaltLabs of Halle, Germany. Participants of the sprint are also invited to join the gocept-23 party at 23rd of September 2023.

This sprint will the last one hosted by gocept as the company will go new ways by the end of the year. To join please register via meetup.com. (After registering for the sprint, you’ll get the party invitation.)

Sprint with Earl Zope in April of 2022

Sprint with Earl Zope at April, 13th 2022 towards Python 3.11 compatibility.

Earl Zope – once in a while – needs some help to feel well again in the ever changing Python 3 wonderland.

Please join a one-day-sprint at April, 13th 2022 organized by gocept to make Earl Zope more comfortable with the current Python 3.10 and the upcoming Python 3.11.

Especially the new upcoming version has some bitter surprises for Earl Zope: The C-code of some of his dependencies will no longer compile without modifications. An example dependency is already ported, so it should be possible to port other ones by the monkey-see-monkey-do principle.

The sprint will be a pure online-sprint. If you want to join please register via meetup.com.

Saltlabs Sprint: last minute information

Earl Zope is now nearly settled down in Python 3 wonderland. On the Zope and Plone sprint from Monday, 1st until Friday, 5th of October 2018 in Halle (Saale), Germany we will work towards the final Zope 4 release aka the final permission for the Python 3 wonderland.

We are currently 33 participants for the sprint. So be prepared for a huge sprint with many interesting people. The Saltlabs have a café (called KOFFIJ) we can use, a big meeting room with big display (aka the Thronsaal) and many smaller rooms including the offices of gocept. So there will be enough room to work in bigger and smaller groups.

To keep the organisational overhead low with this amount of participants, we plan to separate in two teams: Zope and Plone. Those teams will organise themselves individually and we will have a short daily meeting after lunch to share the status in a condensed manner with the other team. Direct communication in case of a difficult problem is, of course, always possible.

We reserved up to one hour after the daily meeting for talks and presentations by you about interesting topics around Zope and Plone, successful migration stories, or something else you want to share with the community. So if you have some interesting slides, please bring them with you and register during the week for a slot.

Our current schedule:

  • Sunday
    • 19:00, there is a table reserved at Grober Gottlieb, so if you’ve already arrived and want some company, you are invited to join.
  • Monday
    • 9:00 Breakfast at KOFFIJ (This is the café in the ground floor of Saltlabs aka the window to the left on the picture above.)
    • 10:00 Welcome at KOFFIJ and start sprinting afterwards
    • 12:30 Lunch
    • 13:30 Sprint planning and introduction for all sprinters at Thronsaal
    • between 15:00 and 17:00 coffee break at KOFFIJ
    • 18:00 Lights out
  • All other days:
    • 8:30 Breakfast
    • 9:00 Standup in the team (Zope, Plone)
    • 12:30 Lunch
    • 13:30 Daily meeting at Thronsaal
    • 14:00 (Lightning) Talks at Thronsaal
    • between 15:00 and 17:00 coffee break at KOFFIJ
    • 18:00 Lights out
  • Tuesday:
    • 11:00 till 17:00 Massages, there will be a list to register on Monday
    • 19:00 social evening at Eigenbaukombinat (local hacker space) with pizza, beer and mate
  • Friday:
    • 13:30 Closing meeting with presentations at Thronsaal
    • 17:00 Lights out

If you cannot make it to the Welcome meeting, ask at KOFFIJ for one of the gocept staff to get a personal introduction.

Parking: As Saltlabs in located in a pedestrian zone, the availability of parking spots is rather low. Please use one of the parking decks nearby.

As organizational tool to coordinate the work, we try to use Github projects this time, as it allows cross-repository tracking of issues.

One last hint: The location of the sprint is Leipziger Str. 70, Halle (Saale), Germany.

Saltlabs Sprint: Zope and Plone sprint in a new location

After Earl Zope II is now nearly relocated to the Python 3 wonderland, gocept will move to a new head quarter in the next months. This is the right time to celebrate with a new sprint, as we have now even more space for sprinters. The new location is called the “Saltlabs”, a place for IT companies in Halle (Saale), Germany.

Sprint information

  • Date: Monday, 1st until Friday, 5th of October 2018
  • Location: Leipziger Str. 70, Halle (Saale), Germany

Sprint topics

This sprint has three main topics:

Create a final Zope 4 release

Before releasing a final version of Zope 4 we want to resolve about at least 40 issues: Some bugs have to be fixed,  some functions have to be polished and documentation has to be written resp. reviewed. On the other hand there is the re-brush of the ZMI using Bootstrap which should be completed beforehand, as it modernizes the ZMI and allows for easier customisation, but might also be backwards incompatible with certain test suites. There is an Etherpad to write down ideas, tasks, wishes and work proposals, which are not currently covered by the issue tracker.

Port Plone to Python 3

The following tasks are currently open and can be fixed at the sprint:

  • successfully run all Plone tests and even the robotframework tests on Python 3
  • Zope 4 lost the WebDAV support: find resp. create a replacement
  • document the WSGI setup and test it in a production ready environment
  • port as many as possible add-ons to Python 3 (e.g. Mosaic and Easyform)
  • work on the Migration of ZODB contents (Data.fs) to Python 3
  • improve the test setup with tox.
  • start to support Python 3.7

Polish Plone 5.2

The upcoming Plone 5.2 release will appreciate some love and care at the following items:

  • new navigation with dropdown and better performance
  • Barceloneta theme: ease the customisation and improve responsiveness
  • parallelise the tests so they run faster
  • remove Archetypes and other obsolete packages

See also the list of topics on plone.org for this sprint.

Organisational Remarks

In order to coordinate the participation for this sprint, we ask you to join us on Meetup. We can then coordinate the catering and requirements for space.

As this sprint will be running longer than usual (five days), it is also possible to join only for a part of the week. As October 3rd is the national holiday, we are trying to organise some social event for those who are interested in having a small break.

For a better overview, please indicate your participation also on this doodle poll.

 

Migrate a Zope ZODB Data.fs to Python 3

TL;DR Use zodbupdate.

Problem

A ZODB Data.fs which was created under Python 2 cannot be opened under Python 3. This is prevented by using a different magic code in the first bytes of the file. This is done on purpose because str has a different meaning for the two Python versions: Under Python 2 a str is a container for characters with an arbitrary encoding (aka bytes​). Python 3 knows str as a text datatype which was called unicode in Python 2. Trying to load a str object in Python 3 which actually contains binary data will fail. It has to be bytes, but bytes is an alias for str  in Python 2 which means Python 2 replaces bytes  with str making is impossible to give Python 3 the class it expects for binary data. A Python 2 str  with an arbitrary encoding will break, too.

Solution

The Data.fs has to be migrated: each str  which actually contains bytes has to be converted into a zodbpickle.binary object which deserialises as bytes under Python 3. The str objects actually containing text have to be decoded to unicode. There are currently two tools which claim that they are able to do such a migration:

  • zodb.py3migrate was already written at Berlin Strategic sprint in 2016, but it was never able to prove that it can do what it claims: At the time when it was written there was no Zope which could run on Python 3. Now as we have Zope 4 running on Python 3 it does not seem to do its conversion job quite well: I was able to migrate a toy database but had to catch an unpickling error.
  • zodbupdate was enriched by a Python 3 migration. A big thank you to Sylvain Viollon and the developers at Minddistrict! It has proven its claims! At the Zope 4 welcome sprint I was able to migrate a Data.fs created on Zope 2.13 running on Python 2 to Zope 4 running on Python 3.

Steps

  1. Migrate your Zope application to Zope 4. (zodbupdate  requires at least ZODB 4 which is not the default ZODB version of Zope 2.13) — For my toy database containing only a file object and an image this was no problem. Zope 4  is starting with such a database. It might show some broken objects because Zope no longer depends on some previous core packages like Products.Sessions. If your application needs those packages you should add them to your Zope environment.
  2. ​zodbupdate has to be installed into the Zope 4 environment so it can access the Python classes. (It has to read the pickles in the ZODB.)
  3. There needs to be an entry_point in setup.py for each package which contains persistent Python classes. The entry point has to be named "zodbupdate.decode" and needs to point to a dictionary mapping paths to  str attributes to a conversion (bytes resp. a specific encoding). For Details see the migration documentation of zodbupdate. I prepared a branch of Zope 4 which contains this configuration dictionary for OFS.Image and OFS.File, see zopefoundation/Zope#285.
  4. Run zodbupdate --pack --convert-py3 on the Data.fs using Python 2.
  5. Copy the Data.fs over to the Zope 4 instance running on Python 3. Data.fs.index will be discarded at the first start. (There is an error message telling that it cannot be read.)
  6. Enjoy the contents of the Data.fs running on Python 3.

Conclusion

It is possible (proven for a toy database) to migrate a Data.fs from Zope 2.13  (Python 2) to Zope 4 (Python 3).

zodbupdate is the way to go. Although it cannot do the migration completely autonomously the developers of Python packages can provide migration configuration in their packages which can be used in the migration step so the configuration has only to be written once.

zodb.py3migrate has an analysis step which shows the attribute names where the str objects are stored. (This could be added to zodbupdate, so do not expect that there will be two tools trying to achieve the same goal.)

mdtools.relstorage contains a relstorage variant of zodbupdate which claims to be much faster on relstorage as it can leverage parallelism.

Open issues

The pull request containing the migration strategy (zopefoundation/Zope#285) has to be extended for the other persistent classes in Zope. There have to be alike changes in all packages providing persistent classes.

Zope is welcome in the Python 3 wonderland!

Earl Zope already got the beta permission to stay in the Python 3 wonderland some months ago. His current objective is to help old friends to come to the Python 3 wonderland and to make new friends. He has to build trust in his will and ability to stay in the Python 3 wonderland.

The Zope-4-Welcome sprint last week was a great opportunity to work towards the final permission for Earl Zope. We were a group of 15 developers from different companies and backgrounds building applications on Zope in various ways.

We accomplished the following goals:

  • There are some old friends of Earl Zope. He thought that he no longer needs them in the Python 3 wonderland but other applications built on Zope need them, so they were pushed towards the new land:
  • Knight RestrictedPython got some love and a new beta release.
  • Earl Zope could help an old friend (a custom Zope 2.13 application) to get prepared to move to the new land.
  • Duchess CMFCore got a beta permission for the Python 3 wonderland including her beloved siblings:
  • Prince Plone is not yet ready to live in the Python 3 wonderland but he is already a welcome guest. It is only a matter of time before he will get an alpha permission:
    • The instance starts and many actions in the UI work pretty well.
    • The test story was brought some steps further so it is possible to start testing Plone under Python 3.
    • Details are described in a Blog post of Philip Bauer.
  • The migration of a toy Data.fs was tested and successfully completed. (Details will follow in another blog post.) The Migration took the following steps:
    • from Zope 2.13 on Python 2.7
    • via Zope 4 on Python 2.7
    • to Zope 4 on Python 3.6
  • The ZMI of Earl Zope got a facelift (Zope#249) which is not complete yet but looks promising.
  • And last but not least Earl Zope himself got the 5th extension of his beta permission: Zope 4.0b5.

Earl Zope says a hearty thank you to all who where involved in this sprint in Halle or remote by coding or providing the resources and time to code.

Welcome in the Python 3 wonderland!

Zope – undo transactions via debug console

If you cannot access the root level in Zope 2 via the browser but you are able to use the debug console you have enough to undo transactions.

Start the debug console:

$ bin/zinstance debug

List the transaction descriptions, user names and ids of the last 10 transactions:

>>> from pprint import pprint
>>> pprint(app._p_jar.db().undoInfo(0, 10))

(0 is the youngest transaction, the first one to show)

Undo some transactions:

>>> import transaction
>>> transaction.get().note(u"Undo transactions: ...")
>>> app._p_jar.db().undoMultiple([id1, id2])
>>> transaction.commit()

(id1, id2 have to be the actual id strings from the transaction listing.)

Be aware:

  • Use a sorted list of transaction ids, start with the id of the youngest transaction you want to undo.
  • You cannot undo transactions those objects were changed again later without undoing those transactions, too.
  • The actual undo is executed when calling transaction.commit(). (This might take some time to execute.)
  • You can check the result of your undo using the transaction log as undo only creates an inverse transaction.

 

This blog post first appeared on icemac15.wordpress.com.

Catching and rendering exceptions

Error handling in Zope 4

TL;DR: You have to write an exception view in file system code which is rendered when an exception occurs.

History

If an exception occurred in Zope 2 the standard_error_message (an object in the ZODB) was rendered. This way the error page could be customised through the web.

When using a WSGI server on Zope 2 the standard_error_message is no longer used. The exceptions have to be handled in a WSGI middleware. (This is a sub-optimal solution as the middleware is not run in the same execution context where the exception occurred.)

Thats why error handling changed again in Zope 4:  Like Zope 3 (aka BlueBream) Zope 4 tries to lookup an exception view if an exception occurs. If the lookup succeeds (aka there is an exception view registered for the current exception) this view is rendered as response. This approach allows different views for different exceptions. The standard_error_message is even gone when installing Zope 4 from scratch.

Continue reading “Catching and rendering exceptions”

Earl Zope II is dead, long live Earl Zope

Zope 4.0b1 released

 

Narrative version

Once upon the time there was Earl Zope II. His lands where threatened by a huge disaster called “Python 2 sunset”. His only chance was to emigrate to the Python 3 wonderland. After a long preparation phase for himself and his courtiers he was able to move to the new land. But the Python 3 wonderland has strict immigration authorities: They only allow “compatible” fellows to get a residency permit. The permission itself has three levels:

  • alpha – To get this level the candidate has to prove that he can breath the Python 3 air and drink the water in this land without getting falling over. Earl Zope II needed a while to acclimate in the Python 3 wonderland as its climate is a bit different from what he was used to.
  • beta – This level requires the the acclimation phase was successful and people could start to rely that the candidate will stay in the new land. Earl Zope II reached this level by proving that all his courtiers – he relies on – are ready for this level.
  • final – To reach this level the candidate needs to prove that he is living in successful relations in Python 3 wonderland. Other inhabitants must be able to trust him and the services he offers. After Earl Zope II now has reached the beta phase he is now able to offer his services and hope that he gets the final level about fall 2018.

Back in his own country Earl Zope II had the official name “Zope2 2.13”. Before the beta phase of his immigration he thought that he would have to change the name to something like “Zope2 4.0b1”. This is looks ugly and some people protested against this name. With the kind and quick help of Baiju Muthukadan Earl Zope II was able to change his official name to “Zope 4.0b1”. Thank you very much Baiju Muthukadan! Earl Zope is very proud of this new shorter name and is heartily thankful for this opportunity.

Technical version

The release of the first Beta version means, that no currently existing features will be removed until the final version. But There will be some new features and many bugfixes.

New Features of Zope version 4.0

  • Support for Python 3.4 up to 3.6: Currently Python 3 can only be used for new projects. There is only an experimental way to convert an existing ZODB from Python 2 to 3 as it is not possible to run the same Data.fs under both versions, see zodb.py3migrate.
  • Zope now by default runs as a WSGI application. The previously used ZServer is still supported but only runs under Python 2.
  • Chameleon based templates are now the default.

Changes

  • The name of the distribution changed from Zope2 to Zope. The previous Zope2 package will remain as a meta package which depends on Zope. This allows packages which require Zope 4.x to depend on Zope instead of Zope2.
  • Removed deprecated code and BBB imports like the Globals package or the internal help system.
  • Some smaller features, bug fixes and security fixes, see the Change log.

Thanks

A big “Thank you!” to all who made this release possible:

  • dedicated people investing time, thoughts and money
  • nice companies allowing their people to participate on Zope sprints
  • the Plone Foundation sponsoring Zope sprints

We had a great Zope 4 Phoenix Sprint helping to raise Zope from the ashes! Thank you to everybody who participated in Halle or from remote.

Roadmap

Beta one of Zope is out:

We need the feedback to adapt Zope to the needs in the wild. The current plan is to create new beta releases once a while after implementing features resp. bug fixes or on demand. The current plan is to release a final 4.0 version in fall of 2018. This should allow software projects built upon Zope to migrate there code before the Python 2 sunset in 2020.