After reanimation we started defibrillation of Zope and … it kinda worked:
On our sprint we got the following things done to help Zope in the Python 3 wonderland:
As grok builds on the ZTK, it is a beneficiary of the reanimation. The following steps have been undertaken to lead it to the Python 3 wonderland:
We have had a discussion about the broader future of Zope:
- There could be more optional dependencies like ZServer in Zope 4.
- The ZMI could be removed altogether because it was not maintained any more for years. It should not be used by applications built on top of Zope anyway. Plone even suggests to block public access to the ZMI.
- There is a road map needed for Zope 4 so the Plone community can pick it up as decided in the Zope 4 PLIP.
- Further discussions should be held on the zope-dev list and in the issue trackers of the Zope foundation on Github. This seems to be the way to get most people involved who are interested in the future of Zope.
Conclusion: Zope is not dead. On the sprint there were nearly 20 people who use Zope for their daily work. Some of them even joined the sprint without the backup of a company in their spare time. Yes, it will need time and effort to keep Zope alive and make it prosper in the Python 3 wonderland, but Zope is still needed and has its place in the audience of web frameworks.
We at gocept will keep Zope as part of our supported technology stack in the projects it fits the purpose and will offer help to others who need to migrate a long term project into the future. We will be at the PyConDE in Munich at the end of October and will be open for questions and further discussions. Do not hesitate to talk to us.
Now we are helping Zope in the Python 3 wonderland: Almost 20 people started with the reanimation of Zope. We are working mostly on porting important dependencies of Zope to Python 3:
Zope 4 is now per default based on WSGI. Thanks to Hanno who invested much time to make the WSGI story of Zope much more streamlined.
We found out that the ZTK (Zope Toolkit) is no longer used in any of the projects using packages of it (Zope, Grok, Bluebream – as it is dead). It should be kept to test compatibility between the packages inside the ZTK.
A little tale
Once upon the time there was the big mighty Zope II. It was one of the leaders in the Python land. It had mighty features like TTW (trough the web) development and its own object oriented database. Many people liked Zope II and trusted it to be the basis for their personal work and the work of their companies.
After some years Zope II got a son named Zope III. It could have been even mightier and more trustworthy than Zope II. But some people argued that Zope III is too complex and too hard to understand and way too different from Zope II which they knew very well.
New, faster and easier to understand frameworks arose beside Zope II and Zope III and the people began to like the new ones more. After many years of power, Zope became weak. Its name got hated by many people in its country. It even got nearly forgotten.
But there are still the people out in the far away kingdoms who built a complete ecosystem on the shoulders of Zope. They cannot completely change the basis of their work within months or even years. They deserve a perspective even for the Python 3 wonderland which is no longer a vague dream but already a future with wide open doors.
Back in reality
We are here to help Zope to find a new home in the Python 3 land and live there happily a great many years. Therefore we scheduled a Sprint named Zope Resurrection Sprint where we will discuss and code on Zope to get it more Python 3 ready.
The big blocker is currently RestrictedPython which relies on modules which are no longer in the Python 3 standard library. But RestrictedPython is an important part of the TTW security concept, so it has to be rewritten from scratch. Folks who think they do not need TTW in their projects can work on a Zope where RestrictedPython is optional.
Besides the Python 3 migration of the code the contents of the application databases using Zope need a conversion, too. On a previous sprint in Berlin, we wrote a possible solution: zodb.py3migrate which needs some more tests with real databases.
Furthermore, there is Zope IV, the successor of Zope II which needs some love to become strong and ready for its adventurous journey as the basis of Plone VI and others.
It is still possible to join the sprint either in Halle (Saale), Germany or remote. We are looking forward to a good time to put up the banner of Zope in the Python land again.
At Berlin Strategic sprint 2016 we developed a tool to analyze a ZODB Filestorage to find Python 2 string objects. If they are in an encoding besides ASCII this is preventing using this Filestorage with Python 3 because of decoding errors arising on loading the pickles.
The tool is even able to convert those strings either to unicode by decoding them using a configurable encoding or convert them to
zodbpickle.binary so Python 3 will read them as
There is documentation of the tool and a repository on GitHub where the code lives.
There are still some questions open:
- Is there already another tool for this analysis/migration?
- Is there already any practical knowledge migrating Filestorage contents to Python 3?
- Do you think such a tool is the right approach to achieve such a migration?
- Is there anyone who wants to try out the tool on a Filestorage of a personal project and share the experiences? (We analyzed two projects where we have access to a Filestorage but we are sure this does not catch all the edge cases.)
We have many test suites which use test layers (e. g. the ones from plone.testing). We want to use py.test and all its fancy features to have a modern test runner. There was no way to convert such tests partly: either you have to port the whole project or you are stuck with the zope.testrunner.
On our Pyramid-Sprint Godefroid Chapelle, Thomas Lotze and me wrote a package which wraps layers as py.test fixtures. The result is gocept.pytestlayer.
For each layer it creates two fixtures: one for the layer setUp/tearDown and one for the testSetUp/testTearDown. The layer fixture is configured for class scope but the plug-in orders the tests and knows about the next test so the layer is only torn down if the next test needs another fixture.
You only have to add a new section to your package buildout and running the test via
detects the layers and displays the needed setup code. See the PyPI-Page of the package for details.
Maybe it is possible to get rid of the fixture setup code, so running tests using layers gets even easier.
Travis-CI is a free hosted continuous integration platform for the open source community. It has a good integration with Github, so each push to a project runs the tests of the project.
gocept.selenium is a python package our company has developed as a test-friendly Python API for Selenium which allows to run tests in a browser.
Travis-CI uses YML-Files to configure the test run. I found only little documentation how to run Selenium tests on Travis-CI. But it is straight forward. The following YML file I took from a personal project of mine. (I simplified it a bit for this blog post.):
- "export DISPLAY=:99.0"
- "sh -e /etc/init.d/xvfb start"
- "wget http://selenium.googlecode.com/files/selenium-server-standalone-2.31.0.jar"
- "java -jar selenium-server-standalone-2.31.0.jar &"
- "export GOCEPT_SELENIUM_BROWSER='*firefox'"
- python bootstrap.py
- Lines 1 – 4: My project is a Python project which currently only runs on Python 2.6. But other Python versions will work as well.
- Lines 5, 6: Firefox needs a running XServer, so we start it first as it takes some seconds to launch. See Travis-CI documentation, too.
- Lines 7, 8: The Selenium server seems not to be installed by default, so get it and launch it.
- Line 9: Tell gocept.selenium to use Firefox to run the tests. (Note: To use the new Webdriver-API in the upcoming version 2 of gocept.selenium you have to set other environment variables.)
- Lines 10 – 14: Install the project and run the tests as usual. (The example uses zc.buildout to do this.)
Note: Although I use the Firefox which is installed by default on the Travis-CI machine, I did not yet find out which version it is.
In a new Pyramid project we used deform to render forms. We did not really like it. (The reasons might be detailed in another post.)
To see if other form libraries do better I gave yafowil a try at our gocept Developer Punsch 3: yafowil comes with written documentation. To get a form in our Pyramid application I had to find out some things which are not so clearly documented:
- Let the project depend on yafowil.webob via setup.py as it contains the necessary WebOb integration.
- Import the
loader from yafowil like below, to allow yafowil to register all its known components (even all the packages in the yafowil.wiget namespace). Otherwise I got strange errors. (The loader symbol is not needed at all in the rest of the code of the form.)
from yafowil import loader
- To get a value displayed in the rendered form use
value keyword parameter in the factory like this:
form['name'] = factory('field:label:text',
value can be a plain value or a function which gets the widget and the runtime data of the widget as parameters.
yafowil.base.Factory.resources_for could be a starting point. (I did not do this yet, so it might be wrong.)
Conclusion: yafowil looks like an interesting framework and after getting a starting point it should be useable in Pyramid, too. Maybe this post can help to ease it a bit.
zope.formlib has the ability to customize the used widget like this:
form_fields = zope.formlib.form.Fields(IKeywords)
form_fields['keywords'].custom_widget = KWSelectWidgetFactory
I do not like this approach for two reasons:
- the widget has to be set manually every time the specific field is used
- there is no easy way to get a display widget if the form or field is not editable for the user
Defining a new schema field and registering the widget for this field seems a bit heavy, so I came up with providing a marker interface on the field:
"""Marker interface to get a special keywords widget."""
keywords = zope.schema.List(
title = _("Edit Keywords"),
value_type = zope.schema.Choice(
I registered the edit widget and display widget for the
IHaveSelectableKeywords interface, so the custom widget does not have to be set in the form like this (edit widget):