Earl Zope II is dead, long live Earl Zope

Zope 4.0b1 released

 

Narrative version

Once upon the time there was Earl Zope II. His lands where threatened by a huge disaster called “Python 2 sunset”. His only chance was to emigrate to the Python 3 wonderland. After a long preparation phase for himself and his courtiers he was able to move to the new land. But the Python 3 wonderland has strict immigration authorities: They only allow “compatible” fellows to get a residency permit. The permission itself has three levels:

  • alpha – To get this level the candidate has to prove that he can breath the Python 3 air and drink the water in this land without getting falling over. Earl Zope II needed a while to acclimate in the Python 3 wonderland as its climate is a bit different from what he was used to.
  • beta – This level requires the the acclimation phase was successful and people could start to rely that the candidate will stay in the new land. Earl Zope II reached this level by proving that all his courtiers – he relies on – are ready for this level.
  • final – To reach this level the candidate needs to prove that he is living in successful relations in Python 3 wonderland. Other inhabitants must be able to trust him and the services he offers. After Earl Zope II now has reached the beta phase he is now able to offer his services and hope that he gets the final level about fall 2018.

Back in his own country Earl Zope II had the official name “Zope2 2.13”. Before the beta phase of his immigration he thought that he would have to change the name to something like “Zope2 4.0b1”. This is looks ugly and some people protested against this name. With the kind and quick help of Baiju Muthukadan Earl Zope II was able to change his official name to “Zope 4.0b1”. Thank you very much Baiju Muthukadan! Earl Zope is very proud of this new shorter name and is heartily thankful for this opportunity.

Technical version

The release of the first Beta version means, that no currently existing features will be removed until the final version. But There will be some new features and many bugfixes.

New Features of Zope version 4.0

  • Support for Python 3.4 up to 3.6: Currently Python 3 can only be used for new projects. There is only an experimental way to convert an existing ZODB from Python 2 to 3 as it is not possible to run the same Data.fs under both versions, see zodb.py3migrate.
  • Zope now by default runs as a WSGI application. The previously used ZServer is still supported but only runs under Python 2.
  • Chameleon based templates are now the default.

Changes

  • The name of the distribution changed from Zope2 to Zope. The previous Zope2 package will remain as a meta package which depends on Zope. This allows packages which require Zope 4.x to depend on Zope instead of Zope2.
  • Removed deprecated code and BBB imports like the Globals package or the internal help system.
  • Some smaller features, bug fixes and security fixes, see the Change log.

Thanks

A big “Thank you!” to all who made this release possible:

  • dedicated people investing time, thoughts and money
  • nice companies allowing their people to participate on Zope sprints
  • the Plone Foundation sponsoring Zope sprints

We had a great Zope 4 Phoenix Sprint helping to raise Zope from the ashes! Thank you to everybody who participated in Halle or from remote.

Roadmap

Beta one of Zope is out:

We need the feedback to adapt Zope to the needs in the wild. The current plan is to create new beta releases once a while after implementing features resp. bug fixes or on demand. The current plan is to release a final 4.0 version in fall of 2018. This should allow software projects built upon Zope to migrate there code before the Python 2 sunset in 2020.

Zope preparing to enter Python 3 wonderland

Once upon the time there was an earl named Zope II. His prophets told him that around the year 2020 suddenly his peaceful country will be devastated: They proclaim that with the “sunset” of  Python 2 as stable pillar of his country, insecurity and pain will invade his borders and hurt everyone living within. There seemed only one possible move forward to escape the disaster: Flee to the Python 3 wonderland, the source of peace and prosperity.

This was not as easy as one might think. Earl Zope II was already an old man. He was in the stable age where changes are no longer easy to achieve and he had many courtiers in his staff which he needed all the day.

The immigration authority of the Python 3 wonderland was very picky about the persons which requested permission to settle down. Many “updates” for Zope II and his staff where required to so they eventually became “compatible” with the new country. Earl Zope II was even forced to change his name to Zope IV to show hat he was ready for Python 3 wonderland.

After much work with the immigration authorities it seemed to be possible for earl Zope IV to enter; only some – but important – formalities were needed before he could be allowed to settle down and call himself a citizen of the Python 3 wonderland.

This is where the tale gets real: We need your help to release a beta version of Zope 4. The hard work seems to be done; but some polish and testing is still required to reach this goal.

We invite you to the Zope 4 Phoenix Sprint to help raising Zope 4 from the ashes! From Wednesday, 13th until Friday, 15th of September 2017 we sprint at the gocept office in Halle (Saale), Germany towards the beta release.

Possible sprint topics could be:

  • Work on issues and pull requests regarding the beta release.
  • Make RestrictedPython beta ready.
  • Work on a Bootstrap of the Zope management interface (ZMI)
  • Port CMF components to Python 3 to test Zope 4 for possible issues
  • Work on Plone to make it ready for Zope 4
  • Try out migration strategies for ZODB content to Python 3.
  • Improve the documentation.

You are heartily invited to join us for the honour of earl Zope IV.

Move documentation from pythonhosted.org to readthedocs.io

Today we migrated the documentation of zodb.py3migrate from pythonhosted.org to zodbpy3migrate.readthedocs.io.

This requires a directory – for this example I name it redir – containing a file named index.html with the following content:

<html>
<head>
 <title>zodb.py3migrate</title>
 <meta http-equiv="refresh"
       content="0; url=http://zodbpy3migrate.rtfd.io" />
</head>
<body>
  <p>
    <a href="http://zodbpy3migrate.rtfd.io">
      Redirect to zodbpy3migrate.rtfd.io
    </a>
  </p>
</body>
</html>

To upload it to pythonhosted.org I called:

py27 setup.py upload_docs --upload-dir=redir

Now pythonhosted.org/zodb.py3migrate points to read the docs.

Credits: The HTML was taken from the Trello board of the avocado-framework.

Zope 2 Resurrection Sprint – Goal accomplished

The sprint days were really busy for Earl Zope II and the people helping him with the Python 3 wonderland immigration authorities.

  • Zope
    • can be installed using Python 3
    • can be started and renders some views
    • has more than 1.700 of more than 2.300 tests running
    • has some optional dependencies left to be ported.
  • To accomplished this by:
    • Complete porting of RestrictedPython, so a first alpha release with the new implementation was released. (This includes about 260 commits, nearly 100 files changed, 9.000 lines of newly written code and 1.000 lines of code deleted.)
    • Port AccessControl to Python 3. This port covers the Python code of the package.
    • Make an alpha release of DocumentTemplate which supports Python 3. It is purely based on Python code. (Thanks Hanno for the porting work from C to Python!)
    • Note: There were problems porting AccessControl and DocumentTemplate to PyPy so we left this out for now. (Volunteers welcome!)

Besides working on Zope there was other ongoing work:

His majesty Earl Zope II says a warm “Thank you!” to all who helped him to start his new live in Python 3 wonderland. There is still enough work to be done so he can live there and having all the comfort and stability of Python 3. See you on the next sprint!

Last call for take off to the Python 3 wonderland

Last information for Zope 2 Resurrection Sprint

We are approaching the Zope 2 Resurrection Sprint and hope that all those who are willing to help earl Zope II on his endeavor to port his realm have already prepared there horses and packed the necessary equipment to arrive in Halle (Saale), Germany.

To help with the preparations we have set up some means of communication:

Etherpad

In the Etherpad we have collected a fundamental set of obstacles that the immigration authority of Python 3 wonderland send to earl Zope II via a mounted messenger. If there are additional problems we need to solve with the immigration or other authorities, feel free to point those out in the pad.

IRC Channel

During the sprint we will have an owl waiting for messages in the #sprint channel on irc.freenode.net, so additional information and questions can be placed there.

General Schedule

In general the gates of the gocept manor in Halle (Saale) are open from 8:00 till 18:00 during the sprint for the squires to help earl Zope II. There will be some refreshments in the morning (8:00 – 9:00) and during lunch time (12:00 – 13:00) in order to keep everyone happy and content.

Apart from that, there will be some fixed points in time to meet:

  • Monday 2017-05-01
    • 19:00 CEST, pre-sprint get-together for early arrivals at Anny Kilkenny. Attention: There will be a bigger political demonstration in Halle which might impact the arrival here, take that into consideration.
  • Tuesday 2017-05-02
    • 9:00 CEST, official welcome and sprint planning afterwards.
    • 16:30-17:30 CEST, Discussion: TBD
    • 18:00 CEST, guided tour through the city of Halle, meeting point
    • 19:30 CEST, dinner and get-together at Wenzels Bierstuben, location, separate bills
  • Wednesday 2017-05-03
    • 9:00 CEST, daily meeting and review
    • 16:30-18:00 CEST, Discussion: TDB
    • 19:00 CEST, BBQ evening in the lovely garden at gocept manor
  • Thursday 2017-05-04
  • Friday 2017-05-05
    • 9:00 CEST, daily meeting and review
    • 13:00 CEST, sprint closing session with review and possibility to present lightning talks of your projects.

We are looking forward to the sprint and hope to overcome the remaining migration problems of earl Zope II.

Zope at the turnpike of the Python 3 wonderland

A little tale

Once upon the time there was an earl named Zope II. He lived happily in a land called Python 2. Since some years there where rumours that a huge disaster would hit the country. The people ironically used to call it “sunset”. Prophets arose and said that 2020 would be the year where this disaster would finally happen.

Zope II got anxious about his future and the future of his descendants. But there were some brave people who liked Zope II and told him of the Python 3 wonderland. A land of eternal joy and happiness without “sunset” disasters and with no problems at all. It seemed like a dream to Zope II – too nice to be true?

After some research it became clear that the Python 3 wonderland was real – not completely as advertised by the people but nice to live in. So Zope II set the goal to settle down in the Python 3 wonderland before the “sunset” disaster would happen.

But this was not as easy as it seemed to be. The immigrant authority told Zope II that he is “not compatible” with the Python 3 wonderland and that he needs “to be ported” to be able to breath the special Python 3 air.

As if this was not enough: As an earl Zope II was not able to migrate without his staff, the many people he relied on for his daily procedures. Some of them have been found to be “ported” already thus they were already “compatible” for the new country. But there was one old, but very important servant of Zope II named RestrictedPython. Zope II could not live without him – but he got told that he never will be “compatible”. The authority even required a “complete rewrite”.

The Python 3 wonderland seemed so near but is was so difficult to reach. But there where so many people who helped Zope II and encouraged him not to give up. Eventually it seemed to be possible for Zope II to get beyond the turnpike into the wonderful land called Python 3.

Back in reality

We are the people who like Zope II and help him to get past the turnpike into Python 3. Since the Alpine City Sprint earlier this month RestrictedPython is no longer a blocker for other packages depending on it to be ported.

Come, join us porting the remaining dependencies of Zope and Zope itself to Python 3. There will be a Zope 2 Resurrection Sprint in May this year in Halle (Saale), Germany at gocept. Join us on site or remote.

Towards RestrictedPython 3

The biggest blocker to port Zope to Python 3 is RestrictedPython.

What is RestrictedPython?

It is a library used by Zope to restrict Python code at instruction level to a bare minimum of trusted functionality. It parses and filters the code for not allowed constructs (such as open()) and adds wrappers around each access on attributes or items. These wrappers can be used by Zope to enforce access control on objects in the ZODB without requiring manual checks in the code.

Why is RestrictedPython needed?

Zope allows writing Python code in the Zope management interface (ZMI) using a web browser (“through the web” aka TTW). This code is stored in the ZODB. The code is executed on the server. It would be dangerous to allow a user to execute arbitrary code with the rights of the web server process. That’s why the code is filtered through RestrictedPython to make sure this approach is not a complete security hole.

RestrictedPython is used in many places of Zope as part of its security model. An experiment on the Zope Resurrection Sprint showed that it would be really hard to create a Zope version which does not need RestrictedPython thus removing the TTW approach.

What is the problem porting RestrictedPython to Python 3?

RestrictedPython relies on the compiler package of the Python standard library. This package no longer exists in Python 3 because it was poorly documented, unmaintained and out of sync with the compiler Python uses itself. (There are whisperings that it was only kept because of Zope.)

Since Python 2.6 there is a new ast module in the Python standard library which is not a direct replacement for compiler. There is no documentation how to replace compiler by ast.

What is the current status?

Several people already worked on various Plone and Zope sprints and mostly in their spare time on a Python 3 branch of RestrictedPython to find out how this package works and to start porting some of its functionality as a proof of concept. It seems to be possible to use ast as the new base for RestrictedPython. Probably the external API of RestrictedPython could be kept stable. But packages using or extending some of the internals of RestrictedPython might need to be updated as well.

What are the next steps?

Many Zope and Plone packages depend on RestrictedPython directly (like AccessControl or Products.ZCatalog) or indirectly (like Products.PythonScripts, plone.app.event or even plone.app.dexterity).

When RestrictedPython has successfully been tested against these packages porting them can start. There is a nice list of all Plone 5.1 dependencies and their status regarding Python 3.

Our goal is to complete porting RestrictedPython by the end of March 2017. It opens up the possibility guiding Zope into the Python 3 wonderland by the end of 2017. This is ambitious, especially if the work is done in spare time besides the daily customer work. You can help us by either contributing PullRequests via Github or review them.

We are planning two Zope sprints in spring and autumn 2017. Furthermore we are grateful for each and every kind of support.