Zope Resurrection Part 1 – Reanimation

Now we are helping Zope in the Python 3 wonderland: Almost 20 people started with the reanimation of Zope. We are working mostly on porting important dependencies of Zope to Python 3:

Zope 4 is now per default based on WSGI. Thanks to Hanno who invested much time to make the WSGI story of Zope much more streamlined.

We found out that the ZTK (Zope Toolkit) is no longer used in any of the projects using packages of it (Zope, Grok, Bluebream – as it is dead). It should be kept to test compatibility between the packages inside the ZTK.

Zope in the Python 3 wonderland

A little tale

Once upon the time there was the big mighty Zope II. It was one of the leaders in the Python land. It had mighty features like TTW (trough the web) development and its own object oriented database. Many people liked Zope II and trusted it to be the basis for their personal work and the work of their companies.

After some years Zope II got a son named Zope III. It could have been even mightier and more trustworthy than Zope II. But some people argued that Zope III is too complex and too hard to understand and way too different from Zope II which they knew very well.

New, faster and easier to understand frameworks arose beside Zope II and Zope III and the people began to like the new ones more. After many years of power, Zope became weak. Its name got hated by many people in its country. It even got nearly forgotten.

But there are still the people out in the far away kingdoms who built a complete ecosystem on the shoulders of Zope. They cannot completely change the basis of their work within months or even years. They deserve a perspective even for the Python 3 wonderland which is no longer a vague dream but already a future with wide open doors.

Back in reality

We are here to help Zope to find a new home in the Python 3 land and live there happily a great many years. Therefore we scheduled a Sprint named Zope Resurrection Sprint where we will discuss and code on Zope to get it more Python 3 ready.

The big blocker is currently RestrictedPython which relies on modules which are no longer in the Python 3 standard library. But RestrictedPython is an important part of the TTW security concept, so it has to be rewritten from scratch. Folks who think they do not need TTW in their projects can work on a Zope where RestrictedPython is optional.

Besides the Python 3 migration of the code the contents of the application databases using Zope need a conversion, too. On a previous sprint in Berlin, we wrote a possible solution: zodb.py3migrate which needs some more tests with real databases.

Furthermore, there is Zope IV, the successor of Zope II which needs some love to become strong and ready for its adventurous journey as the basis of Plone VI and others.

It is still possible to join the sprint either in Halle (Saale), Germany or remote. We are looking forward to a good time to put up the banner of Zope in the Python land again.

zodb.py3migrate: Migrate an existing ZODB Data.fs to be used with Python 3

At Berlin Strategic sprint 2016 we developed a tool to analyze a ZODB Filestorage to find Python 2 string objects. If they are in an encoding besides ASCII this is preventing using this Filestorage with Python 3 because of decoding errors arising on loading the pickles.
The tool is even able to convert those strings either to unicode by decoding them using a configurable encoding or convert them to zodbpickle.binary so Python 3 will read them as bytes.
There is documentation of the tool and a repository on GitHub where the code lives.

There are still some questions open:

  • Is there already another tool for this analysis/migration?
  • Is there already any practical knowledge migrating Filestorage contents to Python 3?
  • Do you think such a tool is the right approach to achieve such a migration?
  • Is there anyone who wants to try out the tool on a Filestorage of a personal project and share the experiences? (We analyzed two projects where we have access to a Filestorage but we are sure this does not catch all the edge cases.)

Run tests using layers with py.test

TL;DR

Long Story

We have many test suites which use test layers (e. g. the ones from plone.testing). We want to use py.test and all its fancy features to have a modern test runner. There was no way to convert such tests partly: either you have to port the whole project or you are stuck with the zope.testrunner.

On our Pyramid-Sprint Godefroid Chapelle, Thomas Lotze and me wrote a package which wraps layers as py.test fixtures. The result is gocept.pytestlayer.

Implementation

For each layer it creates two fixtures: one for the layer setUp/tearDown and one for the testSetUp/testTearDown. The layer fixture is configured for class scope but the plug-in orders the tests and knows about the next test so the layer is only torn down if the next test needs another fixture.

Usage

You only have to add a new section to your package buildout and running the test via

bin/py.test -x

detects the layers and displays the needed setup code. See the PyPI-Page of the package for details.

Future

Maybe it is possible to get rid of the fixture setup code, so running tests using layers gets even easier.

Running tests using gocept.selenium on Travis-CI

Travis-CI is a free hosted continuous integration platform for the open source community. It has a good integration with Github, so each push to a project runs the tests  of the project.

gocept.selenium is a python package our company has developed as a test-friendly Python API for Selenium which allows to run tests in a browser.

Travis-CI uses YML-Files to configure the test run. I found only little documentation how to run Selenium tests on Travis-CI. But it is straight forward. The following YML file I took from a personal project of mine. (I simplified it a bit for this blog post.):

language: python
python:
  - 2.6
before_install:
  - "export DISPLAY=:99.0"
  - "sh -e /etc/init.d/xvfb start"
  - "wget http://selenium.googlecode.com/files/selenium-server-standalone-2.31.0.jar"
  - "java -jar selenium-server-standalone-2.31.0.jar &"
  - "export GOCEPT_SELENIUM_BROWSER='*firefox'"
install:
  - python bootstrap.py
  - bin/buildout
script:
  - bin/test

Explanation:

  • Lines 1 – 4: My project is a Python project which currently only runs on Python 2.6. But other Python versions will work as well.
  • Lines 5, 6: Firefox needs a running XServer, so we start it first as it takes some seconds to launch. See Travis-CI documentation, too.
  • Lines  7, 8: The Selenium server seems not to be installed by default, so get it and launch it.
  • Line 9: Tell gocept.selenium to use Firefox to run the tests. (Note: To use the new Webdriver-API in the upcoming version 2 of gocept.selenium you have to set other environment variables.)
  • Lines 10 – 14: Install the project and run the tests as usual. (The example uses zc.buildout to do this.)

Note: Although I use the Firefox which is installed by default on the Travis-CI machine, I did not yet find out which version it is.

yafowil in a Pyramid project

In a new Pyramid project we used deform to render forms. We did not really like it. (The reasons might be detailed in another post.)

To see if other form libraries do better I gave yafowil a try at our gocept Developer Punsch 3: yafowil comes with written documentation. To get a form in our Pyramid application I had to find out some things which are not so clearly documented:

  • Let the project depend on yafowil.webob via setup.py as it contains the necessary WebOb integration.
  • Import the loader from yafowil like below, to allow yafowil to register all its known components (even all the packages in the yafowil.wiget namespace). Otherwise I got strange errors. (The loader symbol is not needed at all in the rest of the code of the form.)
from yafowil import loader
  • To get a value displayed in the rendered form use value keyword parameter in the factory like this:
form['name'] = factory('field:label:text',
                       props=dict(label=u'name', required=True),
                       value=value_getter)

value can be a plain value or a function which gets the widget and the runtime data of the widget as parameters.

  • Some widgets need JavaScript-Libraries. The integration with Pyramid or Fanstatic is not part of the framework. yafowil.base.Factory.resources_for could be a starting point. (I did not do this yet, so it might be wrong.)

Conclusion: yafowil looks like an interesting framework and after getting a starting point it should be useable in Pyramid, too. Maybe this post can help to ease it a bit.

Custom widgets in zope.formlib

zope.formlib has the ability to customize the used widget like this:

class KeywordsManagementForm(five.formlib.formbase.SubPageForm):
    form_fields = zope.formlib.form.Fields(IKeywords)
    form_fields['keywords'].custom_widget = KWSelectWidgetFactory

I do not like this approach for two reasons:

  • the widget has to be set manually every time the specific field is used
  • there is no easy way to get a display widget if the form or field is not editable for the user

Defining a new schema field and registering the widget for this field seems a bit heavy, so I came up with providing a marker interface on the field:

class IHaveSelectableKeywords(zope.interface.Interface):
    """Marker interface to get a special keywords widget."""

class IKeywords(zope.interface.Interface):
    keywords = zope.schema.List(
        title = _("Edit Keywords"),
        value_type = zope.schema.Choice(
            vocabulary=u"uc.keywords.Keywords"))
    zope.interface.alsoProvides(keywords, IHaveSelectableKeywords)

I registered the edit widget and display widget for the IHaveSelectableKeywords interface, so the custom widget does not have to be set in the form like this (edit widget):

<adapter
   for=".IHaveSelectableKeywords
        zope.publisher.interfaces.browser.IBrowserRequest"
 provides="zope.app.form.browser.interfaces.ISimpleInputWidget"
 factory=".KWSelectWidgetFactory"
 permission="zope.Public" />