Earl Zope is now nearly settled down in Python 3 wonderland. On the Zope and Plone sprint from Monday, 1st until Friday, 5th of October 2018 in Halle (Saale), Germany we will work towards the final Zope 4 release aka the final permission for the Python 3 wonderland.
We are currently 33 participants for the sprint. So be prepared for a huge sprint with many interesting people. The Saltlabs have a café (called KOFFIJ) we can use, a big meeting room with big display (aka the Thronsaal) and many smaller rooms including the offices of gocept. So there will be enough room to work in bigger and smaller groups.
To keep the organisational overhead low with this amount of participants, we plan to separate in two teams: Zope and Plone. Those teams will organise themselves individually and we will have a short daily meeting after lunch to share the status in a condensed manner with the other team. Direct communication in case of a difficult problem is, of course, always possible.
We reserved up to one hour after the daily meeting for talks and presentations by you about interesting topics around Zope and Plone, successful migration stories, or something else you want to share with the community. So if you have some interesting slides, please bring them with you and register during the week for a slot.
Our current schedule:
- 19:00, there is a table reserved at Grober Gottlieb, so if you’ve already arrived and want some company, you are invited to join.
- 9:00 Breakfast at KOFFIJ (This is the café in the ground floor of Saltlabs aka the window to the left on the picture above.)
- 10:00 Welcome at KOFFIJ and start sprinting afterwards
- 12:30 Lunch
- 13:30 Sprint planning and introduction for all sprinters at Thronsaal
- between 15:00 and 17:00 coffee break at KOFFIJ
- 18:00 Lights out
- All other days:
- 8:30 Breakfast
- 9:00 Standup in the team (Zope, Plone)
- 12:30 Lunch
- 13:30 Daily meeting at Thronsaal
- 14:00 (Lightning) Talks at Thronsaal
- between 15:00 and 17:00 coffee break at KOFFIJ
- 18:00 Lights out
- 11:00 till 17:00 Massages, there will be a list to register on Monday
- 19:00 social evening at Eigenbaukombinat (local hacker space) with pizza, beer and mate
- 13:30 Closing meeting with presentations at Thronsaal
- 17:00 Lights out
If you cannot make it to the Welcome meeting, ask at KOFFIJ for one of the gocept staff to get a personal introduction.
Parking: As Saltlabs in located in a pedestrian zone, the availability of parking spots is rather low. Please use one of the parking decks nearby.
As organizational tool to coordinate the work, we try to use Github projects this time, as it allows cross-repository tracking of issues.
One last hint: The location of the sprint is Leipziger Str. 70, Halle (Saale), Germany.
After Earl Zope II is now nearly relocated to the Python 3 wonderland, gocept will move to a new head quarter in the next months. This is the right time to celebrate with a new sprint, as we have now even more space for sprinters. The new location is called the “Saltlabs”, a place for IT companies in Halle (Saale), Germany.
- Date: Monday, 1st until Friday, 5th of October 2018
- Location: Leipziger Str. 70, Halle (Saale), Germany
This sprint has three main topics:
Create a final Zope 4 release
Before releasing a final version of Zope 4 we want to resolve about at least 40 issues: Some bugs have to be fixed, some functions have to be polished and documentation has to be written resp. reviewed. On the other hand there is the re-brush of the ZMI using Bootstrap which should be completed beforehand, as it modernizes the ZMI and allows for easier customisation, but might also be backwards incompatible with certain test suites. There is an Etherpad to write down ideas, tasks, wishes and work proposals, which are not currently covered by the issue tracker.
Port Plone to Python 3
The following tasks are currently open and can be fixed at the sprint:
- successfully run all Plone tests and even the robotframework tests on Python 3
- Zope 4 lost the WebDAV support: find resp. create a replacement
- document the WSGI setup and test it in a production ready environment
- port as many as possible add-ons to Python 3 (e.g. Mosaic and Easyform)
- work on the Migration of ZODB contents (
Data.fs) to Python 3
- improve the test setup with
- start to support Python 3.7
Polish Plone 5.2
The upcoming Plone 5.2 release will appreciate some love and care at the following items:
- new navigation with dropdown and better performance
- Barceloneta theme: ease the customisation and improve responsiveness
- parallelise the tests so they run faster
- remove Archetypes and other obsolete packages
See also the list of topics on plone.org for this sprint.
In order to coordinate the participation for this sprint, we ask you to join us on Meetup. We can then coordinate the catering and requirements for space.
As this sprint will be running longer than usual (five days), it is also possible to join only for a part of the week. As October 3rd is the national holiday, we are trying to organise some social event for those who are interested in having a small break.
For a better overview, please indicate your participation also on this doodle poll.
Earl Zope already got the beta permission to stay in the Python 3 wonderland some months ago. His current objective is to help old friends to come to the Python 3 wonderland and to make new friends. He has to build trust in his will and ability to stay in the Python 3 wonderland.
The Zope-4-Welcome sprint last week was a great opportunity to work towards the final permission for Earl Zope. We were a group of 15 developers from different companies and backgrounds building applications on Zope in various ways.
We accomplished the following goals:
- There are some old friends of Earl Zope. He thought that he no longer needs them in the Python 3 wonderland but other applications built on Zope need them, so they were pushed towards the new land:
- Knight RestrictedPython got some love and a new beta release.
- Earl Zope could help an old friend (a custom Zope 2.13 application) to get prepared to move to the new land.
- Duchess CMFCore got a beta permission for the Python 3 wonderland including her beloved siblings:
- Prince Plone is not yet ready to live in the Python 3 wonderland but he is already a welcome guest. It is only a matter of time before he will get an alpha permission:
- The instance starts and many actions in the UI work pretty well.
- The test story was brought some steps further so it is possible to start testing Plone under Python 3.
- Details are described in a Blog post of Philip Bauer.
- The migration of a toy
Data.fs was tested and successfully completed. (Details will follow in another blog post.) The Migration took the following steps:
- from Zope 2.13 on Python 2.7
- via Zope 4 on Python 2.7
- to Zope 4 on Python 3.6
- The ZMI of Earl Zope got a facelift (Zope#249) which is not complete yet but looks promising.
- And last but not least Earl Zope himself got the 5th extension of his beta permission: Zope 4.0b5.
Earl Zope says a hearty thank you to all who where involved in this sprint in Halle or remote by coding or providing the resources and time to code.
Welcome in the Python 3 wonderland!
Once upon the time there was Earl Zope II. A wise guy was telling him that his world will come to an end. He found out that this was true that he had only some years to prepare to immigrate to the Python 3 wonderland.
His preparation was successful: He got past the strict immigration check and has now a beta permission to stay in the Python 3 wonderland. Earl Zope really likes his new home, but he is missing some friends. Most of them are still at the border of Python 3 wonderland and have to go through the immigration process. Earl Zope would be pleased to offer the same service to his friends as in the old Python 2 land.
To get the final permission to stay in Python 3 wonderland Earl Zope needs to build up trust in his abilities and his stability by other inhabitants and old friends. Therefore, he has to show, that he can work with old and new friends in the Python 3 wonderland.
Zope 4 Welcome sprint – 16th until 18th May 2018
Earl Zope invites you to join a sprint with some helpful people to welcome Zope 4 and his friends in Python 3 wonderland. This means: Bring in your Zope 2 based application and we look together how to port it to Zope 4 or even Python 3. That’s why we call the sprint “Zope 4 Welcome sprint”. You can also help by posting issues or even pull requests about your migration attempts.
Additionally we look forward to work on an improved version of the ZMI (Zope Management Interface) and to fix some bugs preventing Earl Zope from getting the final permission.
The sprint will be located in Halle (Saale), Germany. We meet there from Wednesday, 16th till Friday, 18th of May 2018. Please join us via Meetup even if you are planning to work from remote. You will find more detailed and updated information about the sprint there, too.
If you cannot access the root level in Zope 2 via the browser but you are able to use the debug console you have enough to undo transactions.
Start the debug console:
$ bin/zinstance debug
List the transaction descriptions, user names and ids of the last 10 transactions:
>>> from pprint import pprint
>>> pprint(app._p_jar.db().undoInfo(0, 10))
(0 is the youngest transaction, the first one to show)
Undo some transactions:
>>> import transaction
>>> transaction.get().note(u"Undo transactions: ...")
>>> app._p_jar.db().undoMultiple([id1, id2])
(id1, id2 have to be the actual id strings from the transaction listing.)
- Use a sorted list of transaction ids, start with the id of the youngest transaction you want to undo.
- You cannot undo transactions those objects were changed again later without undoing those transactions, too.
- The actual undo is executed when calling
transaction.commit(). (This might take some time to execute.)
- You can check the result of your undo using the transaction log as undo only creates an inverse transaction.
This blog post first appeared on icemac15.wordpress.com.
Beta-Testing Zope 4 together with PerFact Innovation
TL;DR: There are some rough edges when migrating an existing Data.fs and needed Python code from Zope 2.13 to Zope 4 but there is nothing what cannot be solved.
The German company PerFact Innovation (www.perfact-innovation.de) has a customer product built on Zope 2.13. The code needed for the customisation to different customers is stored in the ZODB. They invited me to a workshop where we looked into the migration story to Zope 4. We used the just released Zope 4.0b2.
After half a day the core functions of the customer product where working including the switch from ZServer to WSGI. There were some rough edges we had to come around. I list them here as a reference for other people who start migrating their code to Zope 4.
We also poke around with a fresh Zope 4 installation on Python 3 to see what this will bring. It was nice to see that a
DTMLMethod calling another one while mixing
unicode no longer leads to strange encoding problems but keeps
Expect most of the bugs and some of the uglinesses listed below to be fixed in Zope 4.0b3.
There are some bugs we could easily hack around in the workshop. Most of them are even fixed by now:
- Some objects cannot be loaded from the ZODB because their classes used to be old-style classes but in Zope 4.0b2 they became new-style classes. Fixed in [zopefoundation/Zope#205]
- Python scripts cannot be called because the compiled code is expected on a different attribute. This requires re-compiling them. Fixed in [zopefoundation/Products.PythonScripts#12]
- Some modules seem no longer allowed to be used in through the web PageTemplates, e. g. Products.PythonScripts.standard. Fixed in [zopefoundation/Zope#209]
- Accessing METAL macros using the
getitem style (e. g. metal:use-macro=“python:here.master.macros[‘foo’]“) is not allowed. Its Python class seems to be missing a security declaration. Reported in [zopefoundation/Zope#210]
- When using Python 3 it is not possible to create a new
Script (Python) in the ZMI because the default script code is not Python 3 compatible. Already fixed in [zopefoundation/Products.PythonScripts#10] awaiting release.
Some things are still a bit raw in Zope 4 but for a beta version it is possible to live with them:
We were facing some of the breaking changes in Zope 4, which will require further development if the customer product should behave and be developed like before:
- Support for WebDAV, XML-RPC and FTP has been moved to the ZServer package. This means a WSGI based installation does not provide support for these protocols. Maybe WSGI middlewares could be written to support them again individually.
products directive is no longer supported in the
zope.conf (only via ZServer). This means that the contents of the
Products directory inside the Zope instance directory have to become Python packages which can be installed via
We were facing some required changes on different levels to get the code running:
- Installation: Some Python packages are no longer dependencies of Zope or they have been extracted to separate packages and Zope does not depend on them. They have to be installed separately when they are needed. These packages are:
- Products code: Zope 4 no longer contains the
Globals module. In Zope 2.13 it only contained imports to keep the backwards compatibility with older Zope releases.
Globals is gone now. To find out how to change your imports look at the Globals module on the 2.13 branch.
- PageTemplates: PageTemplates are now rendered using Chameleon. This prevents you from using
-- in HTML comments as it is not allowed there. (See also the HTML comments document of the WebDesignGroup.)
- Error handling:
standard_error_message does not exist any more in Zope 4. It only supports exception views which have to be written in file system code. See Catching and rendering exceptions as a tutorial how to create exception views and even a hack to restore using
It is not impossible to migrate an existing Data.fs to Zope 4 (while staying on Python 2). It is even easier if the through the web approach was used to create the Data.fs as only very little code has to be modified that way.
Please test your applications on Zope 4 during the beta phase (until fall 2018) so we will get a final release which will be ready for production usage. If you have questions we are here to help.
TL;DR: You have to write an exception view in file system code which is rendered when an exception occurs.
If an exception occurred in Zope 2 the
standard_error_message (an object in the ZODB) was rendered. This way the error page could be customised through the web.
When using a WSGI server on Zope 2 the
standard_error_message is no longer used. The exceptions have to be handled in a WSGI middleware. (This is a sub-optimal solution as the middleware is not run in the same execution context where the exception occurred.)
Thats why error handling changed again in Zope 4: Like Zope 3 (aka BlueBream) Zope 4 tries to lookup an exception view if an exception occurs. If the lookup succeeds (aka there is an exception view registered for the current exception) this view is rendered as response. This approach allows different views for different exceptions. The
standard_error_message is even gone when installing Zope 4 from scratch.
Continue reading “Catching and rendering exceptions”
Once upon the time there was Earl Zope II. His lands where threatened by a huge disaster called “Python 2 sunset”. His only chance was to emigrate to the Python 3 wonderland. After a long preparation phase for himself and his courtiers he was able to move to the new land. But the Python 3 wonderland has strict immigration authorities: They only allow “compatible” fellows to get a residency permit. The permission itself has three levels:
- alpha – To get this level the candidate has to prove that he can breath the Python 3 air and drink the water in this land without getting falling over. Earl Zope II needed a while to acclimate in the Python 3 wonderland as its climate is a bit different from what he was used to.
- beta – This level requires the the acclimation phase was successful and people could start to rely that the candidate will stay in the new land. Earl Zope II reached this level by proving that all his courtiers – he relies on – are ready for this level.
- final – To reach this level the candidate needs to prove that he is living in successful relations in Python 3 wonderland. Other inhabitants must be able to trust him and the services he offers. After Earl Zope II now has reached the beta phase he is now able to offer his services and hope that he gets the final level about fall 2018.
Back in his own country Earl Zope II had the official name “Zope2 2.13”. Before the beta phase of his immigration he thought that he would have to change the name to something like “Zope2 4.0b1”. This is looks ugly and some people protested against this name. With the kind and quick help of Baiju Muthukadan Earl Zope II was able to change his official name to “Zope 4.0b1”. Thank you very much Baiju Muthukadan! Earl Zope is very proud of this new shorter name and is heartily thankful for this opportunity.
The release of the first Beta version means, that no currently existing features will be removed until the final version. But There will be some new features and many bugfixes.
New Features of Zope version 4.0
- Support for Python 3.4 up to 3.6: Currently Python 3 can only be used for new projects. There is only an experimental way to convert an existing ZODB from Python 2 to 3 as it is not possible to run the same
Data.fs under both versions, see zodb.py3migrate.
- Zope now by default runs as a WSGI application. The previously used
ZServer is still supported but only runs under Python 2.
- Chameleon based templates are now the default.
- The name of the distribution changed from
Zope. The previous
Zope2 package will remain as a meta package which depends on
Zope. This allows packages which require Zope 4.x to depend on
Zope instead of
- Removed deprecated code and BBB imports like the
Globals package or the internal help system.
- Some smaller features, bug fixes and security fixes, see the Change log.
A big “Thank you!” to all who made this release possible:
- dedicated people investing time, thoughts and money
- nice companies allowing their people to participate on Zope sprints
- the Plone Foundation sponsoring Zope sprints
We had a great Zope 4 Phoenix Sprint helping to raise Zope from the ashes! Thank you to everybody who participated in Halle or from remote.
Beta one of Zope is out:
We need the feedback to adapt Zope to the needs in the wild. The current plan is to create new beta releases once a while after implementing features resp. bug fixes or on demand. The current plan is to release a final 4.0 version in fall of 2018. This should allow software projects built upon Zope to migrate there code before the Python 2 sunset in 2020.
Once upon the time there was an earl named Zope II. His prophets told him that around the year 2020 suddenly his peaceful country will be devastated: They proclaim that with the “sunset” of Python 2 as stable pillar of his country, insecurity and pain will invade his borders and hurt everyone living within. There seemed only one possible move forward to escape the disaster: Flee to the Python 3 wonderland, the source of peace and prosperity.
This was not as easy as one might think. Earl Zope II was already an old man. He was in the stable age where changes are no longer easy to achieve and he had many courtiers in his staff which he needed all the day.
The immigration authority of the Python 3 wonderland was very picky about the persons which requested permission to settle down. Many “updates” for Zope II and his staff where required to so they eventually became “compatible” with the new country. Earl Zope II was even forced to change his name to Zope IV to show hat he was ready for Python 3 wonderland.
After much work with the immigration authorities it seemed to be possible for earl Zope IV to enter; only some – but important – formalities were needed before he could be allowed to settle down and call himself a citizen of the Python 3 wonderland.
This is where the tale gets real: We need your help to release a beta version of Zope 4. The hard work seems to be done; but some polish and testing is still required to reach this goal.
We invite you to the Zope 4 Phoenix Sprint to help raising Zope 4 from the ashes! From Wednesday, 13th until Friday, 15th of September 2017 we sprint at the gocept office in Halle (Saale), Germany towards the beta release.
Possible sprint topics could be:
- Work on issues and pull requests regarding the beta release.
- Make RestrictedPython beta ready.
- Work on a Bootstrap of the Zope management interface (ZMI)
- Port CMF components to Python 3 to test Zope 4 for possible issues
- Work on Plone to make it ready for Zope 4
- Try out migration strategies for ZODB content to Python 3.
- Improve the documentation.
You are heartily invited to join us for the honour of earl Zope IV.
The sprint days were really busy for Earl Zope II and the people helping him with the Python 3 wonderland immigration authorities.
- can be installed using Python 3
- can be started and renders some views
- has more than 1.700 of more than 2.300 tests running
- has some optional dependencies left to be ported.
- To accomplished this by:
- Complete porting of RestrictedPython, so a first alpha release with the new implementation was released. (This includes about 260 commits, nearly 100 files changed, 9.000 lines of newly written code and 1.000 lines of code deleted.)
- Port AccessControl to Python 3. This port covers the Python code of the package.
- Make an alpha release of DocumentTemplate which supports Python 3. It is purely based on Python code. (Thanks Hanno for the porting work from C to Python!)
- Note: There were problems porting AccessControl and DocumentTemplate to PyPy so we left this out for now. (Volunteers welcome!)
Besides working on Zope there was other ongoing work:
- CMFCore, DCWorkflow, GenericSetup now run on Zope 4 using Python 2.
- Even most tests of CMFDefault are now running on Zope 4 – whereas it runs successfully in the browser.
- Some Plone packages were ported to Python 3:
His majesty Earl Zope II says a warm “Thank you!” to all who helped him to start his new live in Python 3 wonderland. There is still enough work to be done so he can live there and having all the comfort and stability of Python 3. See you on the next sprint!