Saltlabs Sprint: last minute information

Earl Zope is now nearly settled down in Python 3 wonderland. On the Zope and Plone sprint from Monday, 1st until Friday, 5th of October 2018 in Halle (Saale), Germany we will work towards the final Zope 4 release aka the final permission for the Python 3 wonderland.

We are currently 33 participants for the sprint. So be prepared for a huge sprint with many interesting people. The Saltlabs have a café (called KOFFIJ) we can use, a big meeting room with big display (aka the Thronsaal) and many smaller rooms including the offices of gocept. So there will be enough room to work in bigger and smaller groups.

To keep the organisational overhead low with this amount of participants, we plan to separate in two teams: Zope and Plone. Those teams will organise themselves individually and we will have a short daily meeting after lunch to share the status in a condensed manner with the other team. Direct communication in case of a difficult problem is, of course, always possible.

We reserved up to one hour after the daily meeting for talks and presentations by you about interesting topics around Zope and Plone, successful migration stories, or something else you want to share with the community. So if you have some interesting slides, please bring them with you and register during the week for a slot.

Our current schedule:

  • Sunday
    • 19:00, there is a table reserved at Grober Gottlieb, so if you’ve already arrived and want some company, you are invited to join.
  • Monday
    • 9:00 Breakfast at KOFFIJ (This is the café in the ground floor of Saltlabs aka the window to the left on the picture above.)
    • 10:00 Welcome at KOFFIJ and start sprinting afterwards
    • 12:30 Lunch
    • 13:30 Sprint planning and introduction for all sprinters at Thronsaal
    • between 15:00 and 17:00 coffee break at KOFFIJ
    • 18:00 Lights out
  • All other days:
    • 8:30 Breakfast
    • 9:00 Standup in the team (Zope, Plone)
    • 12:30 Lunch
    • 13:30 Daily meeting at Thronsaal
    • 14:00 (Lightning) Talks at Thronsaal
    • between 15:00 and 17:00 coffee break at KOFFIJ
    • 18:00 Lights out
  • Tuesday:
    • 11:00 till 17:00 Massages, there will be a list to register on Monday
    • 19:00 social evening at Eigenbaukombinat (local hacker space) with pizza, beer and mate
  • Friday:
    • 13:30 Closing meeting with presentations at Thronsaal
    • 17:00 Lights out

If you cannot make it to the Welcome meeting, ask at KOFFIJ for one of the gocept staff to get a personal introduction.

Parking: As Saltlabs in located in a pedestrian zone, the availability of parking spots is rather low. Please use one of the parking decks nearby.

As organizational tool to coordinate the work, we try to use Github projects this time, as it allows cross-repository tracking of issues.

One last hint: The location of the sprint is Leipziger Str. 70, Halle (Saale), Germany.

Saltlabs Sprint: Zope and Plone sprint in a new location

After Earl Zope II is now nearly relocated to the Python 3 wonderland, gocept will move to a new head quarter in the next months. This is the right time to celebrate with a new sprint, as we have now even more space for sprinters. The new location is called the “Saltlabs”, a place for IT companies in Halle (Saale), Germany.

Sprint information

  • Date: Monday, 1st until Friday, 5th of October 2018
  • Location: Leipziger Str. 70, Halle (Saale), Germany

Sprint topics

This sprint has three main topics:

Create a final Zope 4 release

Before releasing a final version of Zope 4 we want to resolve about at least 40 issues: Some bugs have to be fixed,  some functions have to be polished and documentation has to be written resp. reviewed. On the other hand there is the re-brush of the ZMI using Bootstrap which should be completed beforehand, as it modernizes the ZMI and allows for easier customisation, but might also be backwards incompatible with certain test suites. There is an Etherpad to write down ideas, tasks, wishes and work proposals, which are not currently covered by the issue tracker.

Port Plone to Python 3

The following tasks are currently open and can be fixed at the sprint:

  • successfully run all Plone tests and even the robotframework tests on Python 3
  • Zope 4 lost the WebDAV support: find resp. create a replacement
  • document the WSGI setup and test it in a production ready environment
  • port as many as possible add-ons to Python 3 (e.g. Mosaic and Easyform)
  • work on the Migration of ZODB contents (Data.fs) to Python 3
  • improve the test setup with tox.
  • start to support Python 3.7

Polish Plone 5.2

The upcoming Plone 5.2 release will appreciate some love and care at the following items:

  • new navigation with dropdown and better performance
  • Barceloneta theme: ease the customisation and improve responsiveness
  • parallelise the tests so they run faster
  • remove Archetypes and other obsolete packages

See also the list of topics on plone.org for this sprint.

Organisational Remarks

In order to coordinate the participation for this sprint, we ask you to join us on Meetup. We can then coordinate the catering and requirements for space.

As this sprint will be running longer than usual (five days), it is also possible to join only for a part of the week. As October 3rd is the national holiday, we are trying to organise some social event for those who are interested in having a small break.

For a better overview, please indicate your participation also on this doodle poll.

 

Zope 2 Resurrection Sprint – Goal accomplished

The sprint days were really busy for Earl Zope II and the people helping him with the Python 3 wonderland immigration authorities.

  • Zope
    • can be installed using Python 3
    • can be started and renders some views
    • has more than 1.700 of more than 2.300 tests running
    • has some optional dependencies left to be ported.
  • To accomplished this by:
    • Complete porting of RestrictedPython, so a first alpha release with the new implementation was released. (This includes about 260 commits, nearly 100 files changed, 9.000 lines of newly written code and 1.000 lines of code deleted.)
    • Port AccessControl to Python 3. This port covers the Python code of the package.
    • Make an alpha release of DocumentTemplate which supports Python 3. It is purely based on Python code. (Thanks Hanno for the porting work from C to Python!)
    • Note: There were problems porting AccessControl and DocumentTemplate to PyPy so we left this out for now. (Volunteers welcome!)

Besides working on Zope there was other ongoing work:

His majesty Earl Zope II says a warm “Thank you!” to all who helped him to start his new live in Python 3 wonderland. There is still enough work to be done so he can live there and having all the comfort and stability of Python 3. See you on the next sprint!

Making CI results visible at Home

We are using a Jenkins server to test our projects continuously. To make the results clearly visible, my colleagues installed a Raspberry Pi to display the testing results on a huge LED strip. As you see in the picture below, the project results are displayed on both sides of the whiteboard. The aggregated result is displayed in the big LED tube on the top.

Picture of our Whiteboard with LEDs on either side displaying test results

This setup was built in 2012, but this year we started to work remotely, i.e. from home. Therefore we have no access to informations which are physically displayed at the office. Some month ago, Daniel Havlik mouthed the idea to build an LED strip “to go”. Since the decreased visibility annoyed me, I was a strong advocate of his idea.

Finally, during the Summer Sprint we held in September, we had the opportunity to work on a project of our choice. Daniel Havlik supplied us with an Arduino Uno and a small LED strip, so we started to work on his idea together with a friend of mine, Oliver Zscheyge.

During the first day, we built a simplistic API for the Arduino to set the color of an LED by providing position and color. To test the API we wrote a small Python script that sent random colors to all LEDs.

At the second day of the sprint, Daniel astonished us, since he had put in some extra hours during the night to build a prototype. It had a wooden frame, colored in black and all. Wow! This was definitely the most “visible” result of the Summer Sprint.

Prototype in a wooden case with LEDs and a USB connector

Later that day, we extended the Python script to retrieve the project states from Jenkins and set the LEDs accordingly. We used an INI-file to define which LED represents which project. For fun we also played a short sound when a project was working and broke or the other way around.

Since one of my co-workers wanted to move back to his home town, he got the prototype. To speak truly, I was a little bit jealous about that. But to my surprise, Daniel built another “LED to go” and gave it to me in October, as a birthday present. Since I am a LARPer, I love the shape of it which resembles a sword.

Remote LED with a Sword-like shape

At some days I like to pronounce that “my sword of Jenkins power is glowing green”, which casts a smile on the faces of my co-workers. I think this was a great project with a very useful result.

If you possess some crafting abilities, you can easily make an “LED to go” yourself. We made our code public at bitbucket/remoteled. It contains the versatile API for the Arduino, as well as the Python script which retrieves data from a Jenkins server. Of course you can build additional clients for your personal needs. If you think it’s useful to others as well, send us a pull request. We are looking forward to it!

With these words I am wishing you all a Merry Christmas and a Happy New Year, since I will not write another post in 2014. See you in January!

Run tests using layers with py.test

TL;DR

Long Story

We have many test suites which use test layers (e. g. the ones from plone.testing). We want to use py.test and all its fancy features to have a modern test runner. There was no way to convert such tests partly: either you have to port the whole project or you are stuck with the zope.testrunner.

On our Pyramid-Sprint Godefroid Chapelle, Thomas Lotze and me wrote a package which wraps layers as py.test fixtures. The result is gocept.pytestlayer.

Implementation

For each layer it creates two fixtures: one for the layer setUp/tearDown and one for the testSetUp/testTearDown. The layer fixture is configured for class scope but the plug-in orders the tests and knows about the next test so the layer is only torn down if the next test needs another fixture.

Usage

You only have to add a new section to your package buildout and running the test via

bin/py.test -x

detects the layers and displays the needed setup code. See the PyPI-Page of the package for details.

Future

Maybe it is possible to get rid of the fixture setup code, so running tests using layers gets even easier.

Running tests using gocept.selenium on Travis-CI

Travis-CI is a free hosted continuous integration platform for the open source community. It has a good integration with Github, so each push to a project runs the tests  of the project.

gocept.selenium is a python package our company has developed as a test-friendly Python API for Selenium which allows to run tests in a browser.

Travis-CI uses YML-Files to configure the test run. I found only little documentation how to run Selenium tests on Travis-CI. But it is straight forward. The following YML file I took from a personal project of mine. (I simplified it a bit for this blog post.):

[code]
language: python
python:
– 2.6
before_install:
– "export DISPLAY=:99.0"
– "sh -e /etc/init.d/xvfb start"
– "wget http://selenium.googlecode.com/files/selenium-server-standalone-2.31.0.jar"
– "java -jar selenium-server-standalone-2.31.0.jar &"
– "export GOCEPT_SELENIUM_BROWSER=’*firefox’"
install:
– python bootstrap.py
– bin/buildout
script:
– bin/test
[/code]

Explanation:

  • Lines 1 – 4: My project is a Python project which currently only runs on Python 2.6. But other Python versions will work as well.
  • Lines 5, 6: Firefox needs a running XServer, so we start it first as it takes some seconds to launch. See Travis-CI documentation, too.
  • Lines  7, 8: The Selenium server seems not to be installed by default, so get it and launch it.
  • Line 9: Tell gocept.selenium to use Firefox to run the tests. (Note: To use the new Webdriver-API in the upcoming version 2 of gocept.selenium you have to set other environment variables.)
  • Lines 10 – 14: Install the project and run the tests as usual. (The example uses zc.buildout to do this.)

Note: Although I use the Firefox which is installed by default on the Travis-CI machine, I did not yet find out which version it is.

News from the toolbox: gocept.selenium and our plans for its future

For a couple of years, we at gocept have been developing a Python library, gocept.selenium, whose goal it is to integrate testing web sites in real browsers with the Python unittest framework. There exist a number of approaches to doing this; when first starting real-browser tests, we opted for using selenium. Back then, it had not been integrated with webdriver yet (more on webdriver below).

There turned out to be multiple aspects to selenium integration: setting up the web server under test, starting a browser to run selenium and pointing it at the server, but also designing a wrapper around the selenium testing API to bring it in line with unittest’s way of defining specialised assertions.

We came up with the gocept.selenium package which includes both a selenese module defining such an API wrapper and a bunch of modules for integration with those web-server frameworks that we happen to use in our work, among them generic WSGI and a number of Zope-related servers. The integration mechanism is implemented in terms of test layers, so all of this requires the Zope test runner to be used. We released a 1.0 version of gocept.selenium in November 2012, marking the selenese API as stable.

The description of the package given so far already indicates two aspects that need yet to be addressed: Firstly, the selenium project is based on webdriver nowadays, with the old selenium implementation being kept for backwards compatibility at the moment. Secondly, collecting all those server integration modules in the same package that implements the actual selenium integration makes for rather complex (albeit optional) package dependencies and poses a maintainability problem.

We have dealt with the latter in December 2012, extracting all those integration modules from gocept.selenium into a new package, gocept.httpserverlayer. From the package’s documentation:
»This package provides an HTTP server for testing your application with normal HTTP clients (e.g. a real browser). This is done using test layers, which are a feature of zope.testrunner. gocept.httpserverlayer uses plone.testing for the test layer implementation, and exposes the following resources (accessible in your test case as self.layer[RESOURCE_NAME]):

  • http_host: The hostname of the HTTP server (Default: localhost)
  • http_port: The port of the HTTP server (Default: 0, which means chosen automatically by the operating system)
  • http_address: hostname:port, convenient to use in URLs (e.g. ‘http://user:password@%s/path’ % self.layer[‘http_address’])

In addition to generic WSGI and static-file serving, the server frameworks supported at this point (i.e. gocept.httpserverlayer 1.0.1) include Zope3/ZTK (both using zope.app.testing and zope.app.wsgi with the latter supporting Grok) as well as Zope2 and Plone (using ZopeTestCase, WSGI or plone.testing.z2).

After the creation of gocept.httpserverlayer, we released the 1.1 series of gocept.selenium which no longer brings its own integration code. For the sake of backwards compatibility, though, it still implements separate TestCase classes for each of the integration flavours.

This leaves webdriver support to be dealt with. Originally, we had hoped to simply sneak it in, having to change very little client code, if any at all. Our plan was to implement the old API (both for test setup and selenese) in terms of webdriver which should allow us to benefit from webdriver immediately, as some issues with the old selenium were causing trouble in our daily work (including the behaviour of type and typeKeys as well as drag-and-drop). We started a branch of gocept.selenium where we switched from integrating legacy selenium to talking to webdriver and changed the selenese implementation to use webdriver commands.

However, it turned out that a number of details couldn’t be completely hidden, and webdriver brought its own share of problems (including, sadly, new issues with drag-and-drop). We tried out our branch in a real project to the point that all tests would pass again, and ended up with a long list of upgrade notes describing incompatibilities, either temporary or not, both causing semantic differences of behaviour and necessitating changes to the test code. We identified a number of pieces of the old selenese API that we wouldn’t bother implementing, and we still had a few large projects that would help discover more things to watch out for.

It became clear that sneaking webdriver into an existing selenium test suite wasn’t the way to get to use it soon. So, instead of continuing to develop the branch and replacing the selenium-based implementation in gocept.selenium 2, we merged the branch now, in such a way that we have two different selenium integrations available at the same time, usable simultaneously in the same project. That way, new browser tests can be added using the webdriver integration layer, and existing tests can be migrated to using webdriver test case by test case, as needed.

We have made alpha releases of gocept.selenium 2 so people may experiment with the webdriver integration. Note that while the current implementation of the test layer (gocept.selenium.webdriver.Layer) contains some code to deal with Firefox, we have successfully run it against Chrome as well. While the integration layer exposes a raw webdriver object as the seleniumrc resource, there is also the WebdriverSeleneseLayer which offers a resource named selenium, which is the old selenese API implemented in terms of webdriver and can be used together with the base layer.

We are currently working towards a stable gocept.selenium 2 release that includes webdriver support at the level described, but at the same time also thinking about how our ideal testing API might be structured in order to integrate with the unittest API concepts but make better use of the object-oriented raw webdriver API than the current selenese does. If you have an interest in using webdriver in conjunction with the Python unittest framework you are very welcome to try out the current state of gocept.selenium 2 and get back to us with ideas and suggestions.

Shutting down an HTTPServer

For integration tests it can be helpful to have a fake HTTP server whose behaviour the tests can control. All necessary building blocks are even included in Python standard library. However, the BaseHTTPServer is surprisingly hard to shut down properly, so that it gives up the socket and everything.

While working on gocept.selenium, we came up with some code that does the trick (together with Jan-Wijbrand Kolman and Jan-Jaap Driessen).

class HTTPServer(BaseHTTPServer.HTTPServer):

    _continue = True

    def serve_until_shutdown(self):
        while self._continue:
            self.handle_request()

    def shutdown(self):
        self._continue = False
        # We fire a last request at the server in order to take it out of the
        # while loop in `self.serve_until_shutdown`.
        try:
            urllib2.urlopen(
                'http://%s:%s/' % (self.server_name, self.server_port))
        except urllib2.URLError:
            # If the server is already shut down, we receive a socket error,
            # which we ignore.
            pass
        self.server_close()

You might use this in a zope.testrunner layer like this:

class SilentRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):

    def log_message(self, format, *args):
        pass


class HTTPServerLayer(object):

    host = 'localhost'

    def setUp(self):
        self.server = None
        self.port = random.randint(30000, 40000)
        self.start_server()

    def start_server(self):
        self.server = HTTPServer((self.host, self.port), SilentRequestHandler)
        self.server_thread = threading.Thread(
            target=self.server.serve_until_shutdown)
        self.server_thread.daemon = True
        self.server_thread.start()
        # Kludge: Wait a little as it sometimes takes a while to get the server
        # started.
        time.sleep(0.25)

    def stop_server(self):
        if self.server is None:
            return
        self.server.shutdown()
        self.server_thread.join()

    def tearDown(self):
        self.stop_server()