Tag: zope

Zope Resurrection Part 2 – Defibrillation

After reanimation we started defibrillation of Zope and … it kinda worked:

On our sprint we got the following things done to help Zope in the Python 3 wonderland:

As grok builds on the ZTK, it is a beneficiary of the reanimation. The following steps have been undertaken to lead it to the Python 3 wonderland:

We have had a discussion about the broader future of Zope:

  • There could be more optional dependencies like ZServer in Zope 4.
  • The ZMI could be removed altogether because it was not maintained any more for years. It should not be used by applications built on top of Zope anyway. Plone even suggests to block public access to the ZMI.
  • There is a road map needed for Zope 4 so the Plone community can pick it up as decided in the Zope 4 PLIP.
  • Further discussions should be held on the zope-dev list and in the issue trackers of the  Zope foundation on Github. This seems to be the way to get most people involved who are interested in the future of Zope.

Conclusion: Zope is not dead. On the sprint there were nearly 20 people who use Zope for their daily work. Some of them even joined the sprint without the backup of a company in their spare time. Yes, it will need time and effort to keep Zope alive and make it prosper in the Python 3 wonderland, but Zope is still needed and has its place in the audience of web frameworks.

We at gocept will keep Zope as part of our supported technology stack in the projects it fits the purpose and will offer help to others who need to migrate a long term project into the future. We will be at the PyConDE in Munich at the end of October and will be open for questions and further discussions. Do not hesitate to talk to us.

 

Zope Resurrection Part 1 – Reanimation

Now we are helping Zope in the Python 3 wonderland: Almost 20 people started with the reanimation of Zope. We are working mostly on porting important dependencies of Zope to Python 3:

Zope 4 is now per default based on WSGI. Thanks to Hanno who invested much time to make the WSGI story of Zope much more streamlined.

We found out that the ZTK (Zope Toolkit) is no longer used in any of the projects using packages of it (Zope, Grok, Bluebream – as it is dead). It should be kept to test compatibility between the packages inside the ZTK.

Last minute information for remote Sprinters for the Zope Resurrection Sprint

As the Zope Resurrection sprint is approaching, it seems useful to share some information on the schedule for the three days in Halle. As we have also some sprinters, who can not join on site, but might want to join remotely, a few key facts might come in handy.

Etherpad

There is the Etherpad, where the current list of topics is collected. Most of the stories we are going to tackle will be based on this list. In case you have something to add, or are interested in a specific topic in particular, it is a good idea to add your thoughts before the start of the sprint.

IRC Channel

During the sprint we will communicate via the #sprint channel on irc.freenode.net, so additional information and questions can be placed there.

Google Hangout

As the sprint is also intended to foster discussions about the future of Zope, we want to encourage as many people as possible to join. Therefore, we have a hangout where we will meet.

Scheduled Discussions

At the moment we plan the following session:

  • Wednesday 2016-09-28 10:00 CEST, duration 1 h: Introduction to WSGI in Zope 4 by Hanno Schlichting (hannosh) + discussion
  • Wednesday 2016-09-28 14:00 CEST, duration 2 h: Discussion about through-the web (TTW) pattern and the underlying RestrictedPython
  • Thursday 2016-09-29 10:00 CEST, duration 0.5 h: Discussion about the necessity and future of ZTK
  • Friday 2016-09-30 14:00 CEST, duration 1 h: Discussion about the Zope 5

So in case you want to contribute remotely to the sprint, please join us on one of the three ways.

 

Zope in the Python 3 wonderland

A little tale

Once upon the time there was the big mighty Zope II. It was one of the leaders in the Python land. It had mighty features like TTW (trough the web) development and its own object oriented database. Many people liked Zope II and trusted it to be the basis for their personal work and the work of their companies.

After some years Zope II got a son named Zope III. It could have been even mightier and more trustworthy than Zope II. But some people argued that Zope III is too complex and too hard to understand and way too different from Zope II which they knew very well.

New, faster and easier to understand frameworks arose beside Zope II and Zope III and the people began to like the new ones more. After many years of power, Zope became weak. Its name got hated by many people in its country. It even got nearly forgotten.

But there are still the people out in the far away kingdoms who built a complete ecosystem on the shoulders of Zope. They cannot completely change the basis of their work within months or even years. They deserve a perspective even for the Python 3 wonderland which is no longer a vague dream but already a future with wide open doors.

Back in reality

We are here to help Zope to find a new home in the Python 3 land and live there happily a great many years. Therefore we scheduled a Sprint named Zope Resurrection Sprint where we will discuss and code on Zope to get it more Python 3 ready.

The big blocker is currently RestrictedPython which relies on modules which are no longer in the Python 3 standard library. But RestrictedPython is an important part of the TTW security concept, so it has to be rewritten from scratch. Folks who think they do not need TTW in their projects can work on a Zope where RestrictedPython is optional.

Besides the Python 3 migration of the code the contents of the application databases using Zope need a conversion, too. On a previous sprint in Berlin, we wrote a possible solution: zodb.py3migrate which needs some more tests with real databases.

Furthermore, there is Zope IV, the successor of Zope II which needs some love to become strong and ready for its adventurous journey as the basis of Plone VI and others.

It is still possible to join the sprint either in Halle (Saale), Germany or remote. We are looking forward to a good time to put up the banner of Zope in the Python land again.

zodb.py3migrate: Migrate an existing ZODB Data.fs to be used with Python 3

At Berlin Strategic sprint 2016 we developed a tool to analyze a ZODB Filestorage to find Python 2 string objects. If they are in an encoding besides ASCII this is preventing using this Filestorage with Python 3 because of decoding errors arising on loading the pickles.
The tool is even able to convert those strings either to unicode by decoding them using a configurable encoding or convert them to zodbpickle.binary so Python 3 will read them as bytes.
There is documentation of the tool and a repository on GitHub where the code lives.

There are still some questions open:

  • Is there already another tool for this analysis/migration?
  • Is there already any practical knowledge migrating Filestorage contents to Python 3?
  • Do you think such a tool is the right approach to achieve such a migration?
  • Is there anyone who wants to try out the tool on a Filestorage of a personal project and share the experiences? (We analyzed two projects where we have access to a Filestorage but we are sure this does not catch all the edge cases.)

Run tests using layers with py.test

TL;DR

Long Story

We have many test suites which use test layers (e. g. the ones from plone.testing). We want to use py.test and all its fancy features to have a modern test runner. There was no way to convert such tests partly: either you have to port the whole project or you are stuck with the zope.testrunner.

On our Pyramid-Sprint Godefroid Chapelle, Thomas Lotze and me wrote a package which wraps layers as py.test fixtures. The result is gocept.pytestlayer.

Implementation

For each layer it creates two fixtures: one for the layer setUp/tearDown and one for the testSetUp/testTearDown. The layer fixture is configured for class scope but the plug-in orders the tests and knows about the next test so the layer is only torn down if the next test needs another fixture.

Usage

You only have to add a new section to your package buildout and running the test via

bin/py.test -x

detects the layers and displays the needed setup code. See the PyPI-Page of the package for details.

Future

Maybe it is possible to get rid of the fixture setup code, so running tests using layers gets even easier.

News from the toolbox: gocept.selenium and our plans for its future

For a couple of years, we at gocept have been developing a Python library, gocept.selenium, whose goal it is to integrate testing web sites in real browsers with the Python unittest framework. There exist a number of approaches to doing this; when first starting real-browser tests, we opted for using selenium. Back then, it had not been integrated with webdriver yet (more on webdriver below).

There turned out to be multiple aspects to selenium integration: setting up the web server under test, starting a browser to run selenium and pointing it at the server, but also designing a wrapper around the selenium testing API to bring it in line with unittest’s way of defining specialised assertions.

We came up with the gocept.selenium package which includes both a selenese module defining such an API wrapper and a bunch of modules for integration with those web-server frameworks that we happen to use in our work, among them generic WSGI and a number of Zope-related servers. The integration mechanism is implemented in terms of test layers, so all of this requires the Zope test runner to be used. We released a 1.0 version of gocept.selenium in November 2012, marking the selenese API as stable.

The description of the package given so far already indicates two aspects that need yet to be addressed: Firstly, the selenium project is based on webdriver nowadays, with the old selenium implementation being kept for backwards compatibility at the moment. Secondly, collecting all those server integration modules in the same package that implements the actual selenium integration makes for rather complex (albeit optional) package dependencies and poses a maintainability problem.

We have dealt with the latter in December 2012, extracting all those integration modules from gocept.selenium into a new package, gocept.httpserverlayer. From the package’s documentation:
»This package provides an HTTP server for testing your application with normal HTTP clients (e.g. a real browser). This is done using test layers, which are a feature of zope.testrunner. gocept.httpserverlayer uses plone.testing for the test layer implementation, and exposes the following resources (accessible in your test case as self.layer[RESOURCE_NAME]):

  • http_host: The hostname of the HTTP server (Default: localhost)
  • http_port: The port of the HTTP server (Default: 0, which means chosen automatically by the operating system)
  • http_address: hostname:port, convenient to use in URLs (e.g. ‘http://user:password@%s/path’ % self.layer[‘http_address’])

In addition to generic WSGI and static-file serving, the server frameworks supported at this point (i.e. gocept.httpserverlayer 1.0.1) include Zope3/ZTK (both using zope.app.testing and zope.app.wsgi with the latter supporting Grok) as well as Zope2 and Plone (using ZopeTestCase, WSGI or plone.testing.z2).

After the creation of gocept.httpserverlayer, we released the 1.1 series of gocept.selenium which no longer brings its own integration code. For the sake of backwards compatibility, though, it still implements separate TestCase classes for each of the integration flavours.

This leaves webdriver support to be dealt with. Originally, we had hoped to simply sneak it in, having to change very little client code, if any at all. Our plan was to implement the old API (both for test setup and selenese) in terms of webdriver which should allow us to benefit from webdriver immediately, as some issues with the old selenium were causing trouble in our daily work (including the behaviour of type and typeKeys as well as drag-and-drop). We started a branch of gocept.selenium where we switched from integrating legacy selenium to talking to webdriver and changed the selenese implementation to use webdriver commands.

However, it turned out that a number of details couldn’t be completely hidden, and webdriver brought its own share of problems (including, sadly, new issues with drag-and-drop). We tried out our branch in a real project to the point that all tests would pass again, and ended up with a long list of upgrade notes describing incompatibilities, either temporary or not, both causing semantic differences of behaviour and necessitating changes to the test code. We identified a number of pieces of the old selenese API that we wouldn’t bother implementing, and we still had a few large projects that would help discover more things to watch out for.

It became clear that sneaking webdriver into an existing selenium test suite wasn’t the way to get to use it soon. So, instead of continuing to develop the branch and replacing the selenium-based implementation in gocept.selenium 2, we merged the branch now, in such a way that we have two different selenium integrations available at the same time, usable simultaneously in the same project. That way, new browser tests can be added using the webdriver integration layer, and existing tests can be migrated to using webdriver test case by test case, as needed.

We have made alpha releases of gocept.selenium 2 so people may experiment with the webdriver integration. Note that while the current implementation of the test layer (gocept.selenium.webdriver.Layer) contains some code to deal with Firefox, we have successfully run it against Chrome as well. While the integration layer exposes a raw webdriver object as the seleniumrc resource, there is also the WebdriverSeleneseLayer which offers a resource named selenium, which is the old selenese API implemented in terms of webdriver and can be used together with the base layer.

We are currently working towards a stable gocept.selenium 2 release that includes webdriver support at the level described, but at the same time also thinking about how our ideal testing API might be structured in order to integrate with the unittest API concepts but make better use of the object-oriented raw webdriver API than the current selenese does. If you have an interest in using webdriver in conjunction with the Python unittest framework you are very welcome to try out the current state of gocept.selenium 2 and get back to us with ideas and suggestions.

Custom widgets in zope.formlib

zope.formlib has the ability to customize the used widget like this:

class KeywordsManagementForm(five.formlib.formbase.SubPageForm):
    form_fields = zope.formlib.form.Fields(IKeywords)
    form_fields['keywords'].custom_widget = KWSelectWidgetFactory

I do not like this approach for two reasons:

  • the widget has to be set manually every time the specific field is used
  • there is no easy way to get a display widget if the form or field is not editable for the user

Defining a new schema field and registering the widget for this field seems a bit heavy, so I came up with providing a marker interface on the field:

class IHaveSelectableKeywords(zope.interface.Interface):
    """Marker interface to get a special keywords widget."""

class IKeywords(zope.interface.Interface):
    keywords = zope.schema.List(
        title = _("Edit Keywords"),
        value_type = zope.schema.Choice(
            vocabulary=u"uc.keywords.Keywords"))
    zope.interface.alsoProvides(keywords, IHaveSelectableKeywords)

I registered the edit widget and display widget for the IHaveSelectableKeywords interface, so the custom widget does not have to be set in the form like this (edit widget):

<adapter
   for=".IHaveSelectableKeywords
        zope.publisher.interfaces.browser.IBrowserRequest"
 provides="zope.app.form.browser.interfaces.ISimpleInputWidget"
 factory=".KWSelectWidgetFactory"
 permission="zope.Public" />

Shutting down an HTTPServer

For integration tests it can be helpful to have a fake HTTP server whose behaviour the tests can control. All necessary building blocks are even included in Python standard library. However, the BaseHTTPServer is surprisingly hard to shut down properly, so that it gives up the socket and everything.

While working on gocept.selenium, we came up with some code that does the trick (together with Jan-Wijbrand Kolman and Jan-Jaap Driessen).

class HTTPServer(BaseHTTPServer.HTTPServer):

    _continue = True

    def serve_until_shutdown(self):
        while self._continue:
            self.handle_request()

    def shutdown(self):
        self._continue = False
        # We fire a last request at the server in order to take it out of the
        # while loop in `self.serve_until_shutdown`.
        try:
            urllib2.urlopen(
                'http://%s:%s/' % (self.server_name, self.server_port))
        except urllib2.URLError:
            # If the server is already shut down, we receive a socket error,
            # which we ignore.
            pass
        self.server_close()

You might use this in a zope.testrunner layer like this:

class SilentRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):

    def log_message(self, format, *args):
        pass


class HTTPServerLayer(object):

    host = 'localhost'

    def setUp(self):
        self.server = None
        self.port = random.randint(30000, 40000)
        self.start_server()

    def start_server(self):
        self.server = HTTPServer((self.host, self.port), SilentRequestHandler)
        self.server_thread = threading.Thread(
            target=self.server.serve_until_shutdown)
        self.server_thread.daemon = True
        self.server_thread.start()
        # Kludge: Wait a little as it sometimes takes a while to get the server
        # started.
        time.sleep(0.25)

    def stop_server(self):
        if self.server is None:
            return
        self.server.shutdown()
        self.server_thread.join()

    def tearDown(self):
        self.stop_server()

 

 

Get a zc.sourcefactory to implement an interface

zc.sourcefactory is very handy to easily create a source (zope.schema.interfaces.IIterableSource to be precise) with corresponding titles and tokens for its contents. Every now and then a source requires an explicit interface. For zc.sourcefactory the following code snippet helps:

class IMySource(zope.schema.interfaces.IIterableSource):
    """my source"""

class MySource(
    zc.sourcefactory.contextual.BasicContextualSourceFactory):
    """The source factory."""

    class source_class(
        zc.sourcefactory.source.FactoredContextualSource):
        """This class is being instanciated by the factory.

        It *must* be called source_class.
        """
        zope.interface.implements(IMySource)

    def getValues(self, context):
        …

Of course it is also possible to declare the source_class separately from the source factory and reference it. But since its sole purpose is to hold an implements declaration, I’m fine with defining it inline.