Sprinting to push Zope to the Python 3 wonderland

Earlier this year there was a sprint in Innsbruck, Austria. We made progress in porting Zope to Python 3 by working on RestrictedPython. After this sprint RestrictedPython no longer seems to be a blocker to port the parts of Zope which rely on RestrictedPython to Python 3.

See the full sprint report on the plone.org website.

We will work further on pushing Zope towards the Python 3 wonderland on the Zope 2 Resurrection Sprint in Halle/Saale, Germany at gocept in the first week of May 2017. You are welcome to  join us on site or remote.

Photo copyright: Christine Baumgartner

Zope at the turnpike of the Python 3 wonderland

A little tale

Once upon the time there was an earl named Zope II. He lived happily in a land called Python 2. Since some years there where rumours that a huge disaster would hit the country. The people ironically used to call it “sunset”. Prophets arose and said that 2020 would be the year where this disaster would finally happen.

Zope II got anxious about his future and the future of his descendants. But there were some brave people who liked Zope II and told him of the Python 3 wonderland. A land of eternal joy and happiness without “sunset” disasters and with no problems at all. It seemed like a dream to Zope II – too nice to be true?

After some research it became clear that the Python 3 wonderland was real – not completely as advertised by the people but nice to live in. So Zope II set the goal to settle down in the Python 3 wonderland before the “sunset” disaster would happen.

But this was not as easy as it seemed to be. The immigrant authority told Zope II that he is “not compatible” with the Python 3 wonderland and that he needs “to be ported” to be able to breath the special Python 3 air.

As if this was not enough: As an earl Zope II was not able to migrate without his staff, the many people he relied on for his daily procedures. Some of them have been found to be “ported” already thus they were already “compatible” for the new country. But there was one old, but very important servant of Zope II named RestrictedPython. Zope II could not live without him – but he got told that he never will be “compatible”. The authority even required a “complete rewrite”.

The Python 3 wonderland seemed so near but is was so difficult to reach. But there where so many people who helped Zope II and encouraged him not to give up. Eventually it seemed to be possible for Zope II to get beyond the turnpike into the wonderful land called Python 3.

Back in reality

We are the people who like Zope II and help him to get past the turnpike into Python 3. Since the Alpine City Sprint earlier this month RestrictedPython is no longer a blocker for other packages depending on it to be ported.

Come, join us porting the remaining dependencies of Zope and Zope itself to Python 3. There will be a Zope 2 Resurrection Sprint in May this year in Halle (Saale), Germany at gocept. Join us on site or remote.

Zope Resurrection Part 2 – Defibrillation

After reanimation we started defibrillation of Zope and … it kinda worked:

On our sprint we got the following things done to help Zope in the Python 3 wonderland:

As grok builds on the ZTK, it is a beneficiary of the reanimation. The following steps have been undertaken to lead it to the Python 3 wonderland:

We have had a discussion about the broader future of Zope:

  • There could be more optional dependencies like ZServer in Zope 4.
  • The ZMI could be removed altogether because it was not maintained any more for years. It should not be used by applications built on top of Zope anyway. Plone even suggests to block public access to the ZMI.
  • There is a road map needed for Zope 4 so the Plone community can pick it up as decided in the Zope 4 PLIP.
  • Further discussions should be held on the zope-dev list and in the issue trackers of the  Zope foundation on Github. This seems to be the way to get most people involved who are interested in the future of Zope.

Conclusion: Zope is not dead. On the sprint there were nearly 20 people who use Zope for their daily work. Some of them even joined the sprint without the backup of a company in their spare time. Yes, it will need time and effort to keep Zope alive and make it prosper in the Python 3 wonderland, but Zope is still needed and has its place in the audience of web frameworks.

We at gocept will keep Zope as part of our supported technology stack in the projects it fits the purpose and will offer help to others who need to migrate a long term project into the future. We will be at the PyConDE in Munich at the end of October and will be open for questions and further discussions. Do not hesitate to talk to us.

 

Last minute information for remote Sprinters for the Zope Resurrection Sprint

As the Zope Resurrection sprint is approaching, it seems useful to share some information on the schedule for the three days in Halle. As we have also some sprinters, who can not join on site, but might want to join remotely, a few key facts might come in handy.

Etherpad

There is the Etherpad, where the current list of topics is collected. Most of the stories we are going to tackle will be based on this list. In case you have something to add, or are interested in a specific topic in particular, it is a good idea to add your thoughts before the start of the sprint.

IRC Channel

During the sprint we will communicate via the #sprint channel on irc.freenode.net, so additional information and questions can be placed there.

Google Hangout

As the sprint is also intended to foster discussions about the future of Zope, we want to encourage as many people as possible to join. Therefore, we have a hangout where we will meet.

Scheduled Discussions

At the moment we plan the following session:

  • Wednesday 2016-09-28 10:00 CEST, duration 1 h: Introduction to WSGI in Zope 4 by Hanno Schlichting (hannosh) + discussion
  • Wednesday 2016-09-28 14:00 CEST, duration 2 h: Discussion about through-the web (TTW) pattern and the underlying RestrictedPython
  • Thursday 2016-09-29 10:00 CEST, duration 0.5 h: Discussion about the necessity and future of ZTK
  • Friday 2016-09-30 14:00 CEST, duration 1 h: Discussion about the Zope 5

So in case you want to contribute remotely to the sprint, please join us on one of the three ways.

 

Zope in the Python 3 wonderland

A little tale

Once upon the time there was the big mighty Zope II. It was one of the leaders in the Python land. It had mighty features like TTW (trough the web) development and its own object oriented database. Many people liked Zope II and trusted it to be the basis for their personal work and the work of their companies.

After some years Zope II got a son named Zope III. It could have been even mightier and more trustworthy than Zope II. But some people argued that Zope III is too complex and too hard to understand and way too different from Zope II which they knew very well.

New, faster and easier to understand frameworks arose beside Zope II and Zope III and the people began to like the new ones more. After many years of power, Zope became weak. Its name got hated by many people in its country. It even got nearly forgotten.

But there are still the people out in the far away kingdoms who built a complete ecosystem on the shoulders of Zope. They cannot completely change the basis of their work within months or even years. They deserve a perspective even for the Python 3 wonderland which is no longer a vague dream but already a future with wide open doors.

Back in reality

We are here to help Zope to find a new home in the Python 3 land and live there happily a great many years. Therefore we scheduled a Sprint named Zope Resurrection Sprint where we will discuss and code on Zope to get it more Python 3 ready.

The big blocker is currently RestrictedPython which relies on modules which are no longer in the Python 3 standard library. But RestrictedPython is an important part of the TTW security concept, so it has to be rewritten from scratch. Folks who think they do not need TTW in their projects can work on a Zope where RestrictedPython is optional.

Besides the Python 3 migration of the code the contents of the application databases using Zope need a conversion, too. On a previous sprint in Berlin, we wrote a possible solution: zodb.py3migrate which needs some more tests with real databases.

Furthermore, there is Zope IV, the successor of Zope II which needs some love to become strong and ready for its adventurous journey as the basis of Plone VI and others.

It is still possible to join the sprint either in Halle (Saale), Germany or remote. We are looking forward to a good time to put up the banner of Zope in the Python land again.

zodb.py3migrate: Migrate an existing ZODB Data.fs to be used with Python 3

At Berlin Strategic sprint 2016 we developed a tool to analyze a ZODB Filestorage to find Python 2 string objects. If they are in an encoding besides ASCII this is preventing using this Filestorage with Python 3 because of decoding errors arising on loading the pickles.
The tool is even able to convert those strings either to unicode by decoding them using a configurable encoding or convert them to zodbpickle.binary so Python 3 will read them as bytes.
There is documentation of the tool and a repository on GitHub where the code lives.

There are still some questions open:

  • Is there already another tool for this analysis/migration?
  • Is there already any practical knowledge migrating Filestorage contents to Python 3?
  • Do you think such a tool is the right approach to achieve such a migration?
  • Is there anyone who wants to try out the tool on a Filestorage of a personal project and share the experiences? (We analyzed two projects where we have access to a Filestorage but we are sure this does not catch all the edge cases.)

Making CI results visible at Home

We are using a Jenkins server to test our projects continuously. To make the results clearly visible, my colleagues installed a Raspberry Pi to display the testing results on a huge LED strip. As you see in the picture below, the project results are displayed on both sides of the whiteboard. The aggregated result is displayed in the big LED tube on the top.

Picture of our Whiteboard with LEDs on either side displaying test results

This setup was built in 2012, but this year we started to work remotely, i.e. from home. Therefore we have no access to informations which are physically displayed at the office. Some month ago, Daniel Havlik mouthed the idea to build an LED strip “to go”. Since the decreased visibility annoyed me, I was a strong advocate of his idea.

Finally, during the Summer Sprint we held in September, we had the opportunity to work on a project of our choice. Daniel Havlik supplied us with an Arduino Uno and a small LED strip, so we started to work on his idea together with a friend of mine, Oliver Zscheyge.

During the first day, we built a simplistic API for the Arduino to set the color of an LED by providing position and color. To test the API we wrote a small Python script that sent random colors to all LEDs.

At the second day of the sprint, Daniel astonished us, since he had put in some extra hours during the night to build a prototype. It had a wooden frame, colored in black and all. Wow! This was definitely the most “visible” result of the Summer Sprint.

Prototype in a wooden case with LEDs and a USB connector

Later that day, we extended the Python script to retrieve the project states from Jenkins and set the LEDs accordingly. We used an INI-file to define which LED represents which project. For fun we also played a short sound when a project was working and broke or the other way around.

Since one of my co-workers wanted to move back to his home town, he got the prototype. To speak truly, I was a little bit jealous about that. But to my surprise, Daniel built another “LED to go” and gave it to me in October, as a birthday present. Since I am a LARPer, I love the shape of it which resembles a sword.

Remote LED with a Sword-like shape

At some days I like to pronounce that “my sword of Jenkins power is glowing green”, which casts a smile on the faces of my co-workers. I think this was a great project with a very useful result.

If you possess some crafting abilities, you can easily make an “LED to go” yourself. We made our code public at bitbucket/remoteled. It contains the versatile API for the Arduino, as well as the Python script which retrieves data from a Jenkins server. Of course you can build additional clients for your personal needs. If you think it’s useful to others as well, send us a pull request. We are looking forward to it!

With these words I am wishing you all a Merry Christmas and a Happy New Year, since I will not write another post in 2014. See you in January!