Towards RestrictedPython 3

The biggest blocker to port Zope to Python 3 is RestrictedPython.

What is RestrictedPython?

It is a library used by Zope to restrict Python code at instruction level to a bare minimum of trusted functionality. It parses and filters the code for not allowed constructs (such as open()) and adds wrappers around each access on attributes or items. These wrappers can be used by Zope to enforce access control on objects in the ZODB without requiring manual checks in the code.

Why is RestrictedPython needed?

Zope allows writing Python code in the Zope management interface (ZMI) using a web browser (“through the web” aka TTW). This code is stored in the ZODB. The code is executed on the server. It would be dangerous to allow a user to execute arbitrary code with the rights of the web server process. That’s why the code is filtered through RestrictedPython to make sure this approach is not a complete security hole.

RestrictedPython is used in many places of Zope as part of its security model. An experiment on the Zope Resurrection Sprint showed that it would be really hard to create a Zope version which does not need RestrictedPython thus removing the TTW approach.

What is the problem porting RestrictedPython to Python 3?

RestrictedPython relies on the compiler package of the Python standard library. This package no longer exists in Python 3 because it was poorly documented, unmaintained and out of sync with the compiler Python uses itself. (There are whisperings that it was only kept because of Zope.)

Since Python 2.6 there is a new ast module in the Python standard library which is not a direct replacement for compiler. There is no documentation how to replace compiler by ast.

What is the current status?

Several people already worked on various Plone and Zope sprints and mostly in their spare time on a Python 3 branch of RestrictedPython to find out how this package works and to start porting some of its functionality as a proof of concept. It seems to be possible to use ast as the new base for RestrictedPython. Probably the external API of RestrictedPython could be kept stable. But packages using or extending some of the internals of RestrictedPython might need to be updated as well.

What are the next steps?

Many Zope and Plone packages depend on RestrictedPython directly (like AccessControl or Products.ZCatalog) or indirectly (like Products.PythonScripts, or even

When RestrictedPython has successfully been tested against these packages porting them can start. There is a nice list of all Plone 5.1 dependencies and their status regarding Python 3.

Our goal is to complete porting RestrictedPython by the end of March 2017. It opens up the possibility guiding Zope into the Python 3 wonderland by the end of 2017. This is ambitious, especially if the work is done in spare time besides the daily customer work. You can help us by either contributing PullRequests via Github or review them.

We are planning two Zope sprints in spring and autumn 2017. Furthermore we are grateful for each and every kind of support.

Zope Resurrection Part 2 – Defibrillation

After reanimation we started defibrillation of Zope and … it kinda worked:

On our sprint we got the following things done to help Zope in the Python 3 wonderland:

As grok builds on the ZTK, it is a beneficiary of the reanimation. The following steps have been undertaken to lead it to the Python 3 wonderland:

We have had a discussion about the broader future of Zope:

  • There could be more optional dependencies like ZServer in Zope 4.
  • The ZMI could be removed altogether because it was not maintained any more for years. It should not be used by applications built on top of Zope anyway. Plone even suggests to block public access to the ZMI.
  • There is a road map needed for Zope 4 so the Plone community can pick it up as decided in the Zope 4 PLIP.
  • Further discussions should be held on the zope-dev list and in the issue trackers of the  Zope foundation on Github. This seems to be the way to get most people involved who are interested in the future of Zope.

Conclusion: Zope is not dead. On the sprint there were nearly 20 people who use Zope for their daily work. Some of them even joined the sprint without the backup of a company in their spare time. Yes, it will need time and effort to keep Zope alive and make it prosper in the Python 3 wonderland, but Zope is still needed and has its place in the audience of web frameworks.

We at gocept will keep Zope as part of our supported technology stack in the projects it fits the purpose and will offer help to others who need to migrate a long term project into the future. We will be at the PyConDE in Munich at the end of October and will be open for questions and further discussions. Do not hesitate to talk to us.


Zope Resurrection Part 1 – Reanimation

Now we are helping Zope in the Python 3 wonderland: Almost 20 people started with the reanimation of Zope. We are working mostly on porting important dependencies of Zope to Python 3:

Zope 4 is now per default based on WSGI. Thanks to Hanno who invested much time to make the WSGI story of Zope much more streamlined.

We found out that the ZTK (Zope Toolkit) is no longer used in any of the projects using packages of it (Zope, Grok, Bluebream – as it is dead). It should be kept to test compatibility between the packages inside the ZTK.

Last minute information for remote Sprinters for the Zope Resurrection Sprint

As the Zope Resurrection sprint is approaching, it seems useful to share some information on the schedule for the three days in Halle. As we have also some sprinters, who can not join on site, but might want to join remotely, a few key facts might come in handy.


There is the Etherpad, where the current list of topics is collected. Most of the stories we are going to tackle will be based on this list. In case you have something to add, or are interested in a specific topic in particular, it is a good idea to add your thoughts before the start of the sprint.

IRC Channel

During the sprint we will communicate via the #sprint channel on, so additional information and questions can be placed there.

Google Hangout

As the sprint is also intended to foster discussions about the future of Zope, we want to encourage as many people as possible to join. Therefore, we have a hangout where we will meet.

Scheduled Discussions

At the moment we plan the following session:

  • Wednesday 2016-09-28 10:00 CEST, duration 1 h: Introduction to WSGI in Zope 4 by Hanno Schlichting (hannosh) + discussion
  • Wednesday 2016-09-28 14:00 CEST, duration 2 h: Discussion about through-the web (TTW) pattern and the underlying RestrictedPython
  • Thursday 2016-09-29 10:00 CEST, duration 0.5 h: Discussion about the necessity and future of ZTK
  • Friday 2016-09-30 14:00 CEST, duration 1 h: Discussion about the Zope 5

So in case you want to contribute remotely to the sprint, please join us on one of the three ways.


Zope in the Python 3 wonderland

A little tale

Once upon the time there was the big mighty Zope II. It was one of the leaders in the Python land. It had mighty features like TTW (trough the web) development and its own object oriented database. Many people liked Zope II and trusted it to be the basis for their personal work and the work of their companies.

After some years Zope II got a son named Zope III. It could have been even mightier and more trustworthy than Zope II. But some people argued that Zope III is too complex and too hard to understand and way too different from Zope II which they knew very well.

New, faster and easier to understand frameworks arose beside Zope II and Zope III and the people began to like the new ones more. After many years of power, Zope became weak. Its name got hated by many people in its country. It even got nearly forgotten.

But there are still the people out in the far away kingdoms who built a complete ecosystem on the shoulders of Zope. They cannot completely change the basis of their work within months or even years. They deserve a perspective even for the Python 3 wonderland which is no longer a vague dream but already a future with wide open doors.

Back in reality

We are here to help Zope to find a new home in the Python 3 land and live there happily a great many years. Therefore we scheduled a Sprint named Zope Resurrection Sprint where we will discuss and code on Zope to get it more Python 3 ready.

The big blocker is currently RestrictedPython which relies on modules which are no longer in the Python 3 standard library. But RestrictedPython is an important part of the TTW security concept, so it has to be rewritten from scratch. Folks who think they do not need TTW in their projects can work on a Zope where RestrictedPython is optional.

Besides the Python 3 migration of the code the contents of the application databases using Zope need a conversion, too. On a previous sprint in Berlin, we wrote a possible solution: zodb.py3migrate which needs some more tests with real databases.

Furthermore, there is Zope IV, the successor of Zope II which needs some love to become strong and ready for its adventurous journey as the basis of Plone VI and others.

It is still possible to join the sprint either in Halle (Saale), Germany or remote. We are looking forward to a good time to put up the banner of Zope in the Python land again.

zodb.py3migrate: Migrate an existing ZODB Data.fs to be used with Python 3

At Berlin Strategic sprint 2016 we developed a tool to analyze a ZODB Filestorage to find Python 2 string objects. If they are in an encoding besides ASCII this is preventing using this Filestorage with Python 3 because of decoding errors arising on loading the pickles.
The tool is even able to convert those strings either to unicode by decoding them using a configurable encoding or convert them to zodbpickle.binary so Python 3 will read them as bytes.
There is documentation of the tool and a repository on GitHub where the code lives.

There are still some questions open:

  • Is there already another tool for this analysis/migration?
  • Is there already any practical knowledge migrating Filestorage contents to Python 3?
  • Do you think such a tool is the right approach to achieve such a migration?
  • Is there anyone who wants to try out the tool on a Filestorage of a personal project and share the experiences? (We analyzed two projects where we have access to a Filestorage but we are sure this does not catch all the edge cases.)

Manage JavaScript dependencies with BowerStatic

Last month I explained how to use Fanstatic to manage JS dependencies. Since we were more and more displeased by using Fanstatic, we recently switched to BowerStatic, the new kid on the block. Since the setup is a bit more complicated and you need more tools to have the same features as you had with Fanstatic, I will describe how to set up the full toolchain. After that, I will shortly summarize the benefits and drawbacks of BowerStatic, so you can decide for yourself if you prefer BowerStatic above Fanstatic or the other way around.

Of course the choices made in the toolchain of BowerStatic are strongly linked to other tools we use. Since we set up large projects using Buildout, we like to use Buildout recipes to solve a problem. Therefore this blogpost is most interesting when you are using Buildout yourself.

How it works

BowerStatic was created by Martijn Faassen, who was a core developer of Fanstatic. We think of BowerStatic as a reimagined version of Fanstatic, avoiding disadvantageos decisions made in the past. One problem of Fanstatic was, that it tried to do too many things: Fetching the files, delivering them to the client and bundling them into one large file.

Therefore BowerStatic has a narrow focus: It only helps to manage which libraries should be delivered to the client. It does not care how the files were retrieved or whether you want to have a bundled resource. To automate these steps as well, you must look for additional tools.

We use the following toolchain:

  • a Buildout recipe for Bower to download all external libraries
  • BowerStatic to deliver JS files to the client
  • a custom Buildout recipe to bundle all files of the project into one large file

Since you need all of these tools to have the same features as Fanstatic, I will give a short description, how to set things up.


Bower is a small tool to download libraries and their dependencies by using a simple description of the library in JSON, which is named bower.json. The description contains a version string, the path to contained files, a list of libraries it depends on and a lot more metadata like author, homepage, keywords and so on. For example:

  "name": "bootstrap",
  "version": "3.3.2",
  "main": [
  "dependencies": {
    "jquery": ">= 1.9.1"

The great thing about Bower is, that you just point to a github repository in case this certain library is not registered on the Bower server. In this case it will check out the trunk of the repository and creates the bower.json all by itself using the commit hash as it’s version. Of course it cannot auto-detect dependencies, which are normally declared in the bower.json.

It’s even better if the repository contains a bower.json file. In this case Bower will only load the files mentioned there and can retrieve dependencies. Therefore it makes no difference if you point Bower to a registered package, e.g. jquery-ui, or if you point it to the github repository of this project, e.g. Of course the version will be different (stable release vs trunk).

To download external dependencies via Bower, we added bowerrecipe to our Buildout configuration. This way we can just list the libraries and their version in the recipe configuration. Bower will automatically download those libraries, as well as their dependencies. The following example will download Bootstrap and jQuery, since Bootstrap depends on jQuery.

recipe = bowerrecipe
packages =
executable = ${buildout:bin-directory}/bower

By default the libraries will be stored inside parts/bower/downloads. The destination can be changed inside the recipe configuration.


BowerStatic uses libraries downloaded via Bower to resolve JS dependencies and delivers all required files to the client. To do so, it must be registered as a WSGI middleware. Compared to Fanstatic, the setup of BowerStatic is a little bit more complicated:

  1. Create an instance of the Bower class
  2. Create a external component collection, which contains all external libraries loaded into the directory parts/bower
  3. Create a local component collection, which will contain your self-written JS files
  4. Set the external collection as the fallback of your local collection, i.e. JS libraries not found in the local collection will be looked up in the external collection
  5. Add your self-written files to the local collection, by reading a bower.json at given path

This may sound like voodoo to you. The documentation of BowerStatic contains some insightful statements why the setup is so complicated. Written in code the setup will look like this:

import bowerstatic
import os
import pkg_resources

bower = bowerstatic.Bower()
external_components = bower.components(
    name='components', path='parts/bower/downloads')
local_components = bower.local_components(
    name='local', component_collection=external_components)

    '', 'resources'), version=None)

Let’s assume in is a bower.json like:

  "name": "custom",
  "version": "0.1",
  "main": [
  "dependencies": {
    "bootstrap": "*"

Then you have the JS library called custom in your local component collection. After handling this setup, you can use BowerStatic similar to Fanstatic, by including JS files inside any view. However, you always need the component collection to do so. local_components.include(self.request.environ, 'custom') will load custom.js, custom.css, the bootstrap library and the jQuery library. You could also write local_components.include(self.request.environ, 'jquery'), which would only load jQuery.


Since we were used to Fanstatics automatism to bundle and minify all resources, we wanted to have something similar with BowerStatic. It is not sure if BowerStatic will ever contain bundling mechanisms, since this may be out of the narrow scope it wants to preserve.

Therefore we build our own Buildout recipe, gocept.recipe.bowerstaticbundler. This recipe will create another Bower library in parts/bower, which contains a minified JS file, a minified CSS file and assets like images and fonts. By using the dependency mechanism of BowerStatic, all JS files are bundled in the right order.

To use the bundled library in production and the separate files in development mode, we build a custom include with an environment variable switch:

def bower_include(environ, name):
    if not os.environ.get('BOWERSTATIC_DEBUG'):
        name = 'bowerstatic_bundle_' + name.replace('.', '_')
    include = local_components.includer(environ)

Of course you must use this include method instead of using the BowerStatic component directly, i.e. replace alo occurences of local_components.include(environ, name) with bower_include(environ, name).

This solution is good enough for ourselves, but may not be ready for your purposes. We have build it with a certain project in mind, so you might come across bugs when using it with different JS libraries. Since we will soon use it in production, we will happily fix bugs you report and merge tested improvements. You can find the code and issue tracker on bitbucket.


The combination of Bower, BowerStatic and a bundling mechanism has the same benefits as Fanstatic, i.e. that you no longer need to copy JS files into your project and link them statically, as well as the benefits of bundled resources to speed up the initial page load.

The biggest contrast of these solutions is how external libraries have to be prepared: With Fanstatic you had to build an integration package for each library and release a version on PyPI for each version of the library. With BowerStatic it’s enough to have a current bower.json inside the library. So both need some kind of work and maintenance.

However, with Fanstatic the assumption was that someone else already built the integration package for you, so that you had to do nothing. As described in the Fanstatic article from last month, this assumption is no longer valid and many JS libraries are not supported or out of date. Updating a foreign integration package consumes time and may not be possible at all, in case the user is no longer active.

With BowerStatic, you can simply point to a git repository and select the version by using a certain commit or tag. As mentioned before, Bower will download the trunk of the repository and generates a simple bower.json automatically. Therefore every JS library that is hosted on github can be used with BowerStatic without additional work. This means no integration packages anymore, yay!

The downside is that you must declare dependencies of those libraries as your dependencies, since Bower cannot detect dependencies when it automatically creates a bower.json from the trunk of the repository. However, if there is a bower.json inside the repository, Bower will use it and to resolve dependencies.

The possibility of Bower to download a github repository comes with the benefit that it is really easy to try out different libraries for the same task. With Fanstatic you had to build an integration package for every experiment or copy the files into your resources. Now you can just point to the github repository and give it a try. Fanstatic always felt like a big hindrance when trying out new libraries.


In my opinion the biggest drawback of BowerStatic is the long toolchain. You have to understand how Bower works, how BowerStatic uses Bower and how to set it up. Building a custom recipe for bundling was a bit time consuming as well.

After about 2 weeks we had aggregated enough wisdom to use the toolchain smoothly. We think of it as an investment in the future, since we were so annoyed by Fanstatic, building one integration package after the other.

Another big downside is, that there is no obvious solution to bundling when using BowerStatic. Of course you could bundle the resources using Grunt or any other approach of your choice. But we wanted a solution that fits into our toolchain, i.e. a solution that works with Python and integrates into Buildout. We did not know of any solutions that matched this criteria, therefore we created the Buildout recipe ourselves. However, building the recipe consumed some time and it is nowhere near “complete”. I hope that there will be a bundling solution that integrates nicely with BowerStatic in the near future. Till then, the Buildout recipe is good enough.


Wow, that was a long article. I hope it was interesting for you to read about BowerStatic. Chances are that you did not know about it before. Setting it up did take some time, but maybe the code snippets above help you to speed things up.

All in all I am really glad that we switched to BowerStatic. After the initial learning investment, you know almost everything that you need to know for the future. Testing brand new JavaScript libraries is a breeze. Even using an unknown library takes no additional effort. Sounds like a win to me.