Tag: sprint

Zope Resurrection Part 2 – Defibrillation

After reanimation we started defibrillation of Zope and … it kinda worked:

On our sprint we got the following things done to help Zope in the Python 3 wonderland:

As grok builds on the ZTK, it is a beneficiary of the reanimation. The following steps have been undertaken to lead it to the Python 3 wonderland:

We have had a discussion about the broader future of Zope:

  • There could be more optional dependencies like ZServer in Zope 4.
  • The ZMI could be removed altogether because it was not maintained any more for years. It should not be used by applications built on top of Zope anyway. Plone even suggests to block public access to the ZMI.
  • There is a road map needed for Zope 4 so the Plone community can pick it up as decided in the Zope 4 PLIP.
  • Further discussions should be held on the zope-dev list and in the issue trackers of the  Zope foundation on Github. This seems to be the way to get most people involved who are interested in the future of Zope.

Conclusion: Zope is not dead. On the sprint there were nearly 20 people who use Zope for their daily work. Some of them even joined the sprint without the backup of a company in their spare time. Yes, it will need time and effort to keep Zope alive and make it prosper in the Python 3 wonderland, but Zope is still needed and has its place in the audience of web frameworks.

We at gocept will keep Zope as part of our supported technology stack in the projects it fits the purpose and will offer help to others who need to migrate a long term project into the future. We will be at the PyConDE in Munich at the end of October and will be open for questions and further discussions. Do not hesitate to talk to us.

 

Last minute information for remote Sprinters for the Zope Resurrection Sprint

As the Zope Resurrection sprint is approaching, it seems useful to share some information on the schedule for the three days in Halle. As we have also some sprinters, who can not join on site, but might want to join remotely, a few key facts might come in handy.

Etherpad

There is the Etherpad, where the current list of topics is collected. Most of the stories we are going to tackle will be based on this list. In case you have something to add, or are interested in a specific topic in particular, it is a good idea to add your thoughts before the start of the sprint.

IRC Channel

During the sprint we will communicate via the #sprint channel on irc.freenode.net, so additional information and questions can be placed there.

Google Hangout

As the sprint is also intended to foster discussions about the future of Zope, we want to encourage as many people as possible to join. Therefore, we have a hangout where we will meet.

Scheduled Discussions

At the moment we plan the following session:

  • Wednesday 2016-09-28 10:00 CEST, duration 1 h: Introduction to WSGI in Zope 4 by Hanno Schlichting (hannosh) + discussion
  • Wednesday 2016-09-28 14:00 CEST, duration 2 h: Discussion about through-the web (TTW) pattern and the underlying RestrictedPython
  • Thursday 2016-09-29 10:00 CEST, duration 0.5 h: Discussion about the necessity and future of ZTK
  • Friday 2016-09-30 14:00 CEST, duration 1 h: Discussion about the Zope 5

So in case you want to contribute remotely to the sprint, please join us on one of the three ways.

 

Zope in the Python 3 wonderland

A little tale

Once upon the time there was the big mighty Zope II. It was one of the leaders in the Python land. It had mighty features like TTW (trough the web) development and its own object oriented database. Many people liked Zope II and trusted it to be the basis for their personal work and the work of their companies.

After some years Zope II got a son named Zope III. It could have been even mightier and more trustworthy than Zope II. But some people argued that Zope III is too complex and too hard to understand and way too different from Zope II which they knew very well.

New, faster and easier to understand frameworks arose beside Zope II and Zope III and the people began to like the new ones more. After many years of power, Zope became weak. Its name got hated by many people in its country. It even got nearly forgotten.

But there are still the people out in the far away kingdoms who built a complete ecosystem on the shoulders of Zope. They cannot completely change the basis of their work within months or even years. They deserve a perspective even for the Python 3 wonderland which is no longer a vague dream but already a future with wide open doors.

Back in reality

We are here to help Zope to find a new home in the Python 3 land and live there happily a great many years. Therefore we scheduled a Sprint named Zope Resurrection Sprint where we will discuss and code on Zope to get it more Python 3 ready.

The big blocker is currently RestrictedPython which relies on modules which are no longer in the Python 3 standard library. But RestrictedPython is an important part of the TTW security concept, so it has to be rewritten from scratch. Folks who think they do not need TTW in their projects can work on a Zope where RestrictedPython is optional.

Besides the Python 3 migration of the code the contents of the application databases using Zope need a conversion, too. On a previous sprint in Berlin, we wrote a possible solution: zodb.py3migrate which needs some more tests with real databases.

Furthermore, there is Zope IV, the successor of Zope II which needs some love to become strong and ready for its adventurous journey as the basis of Plone VI and others.

It is still possible to join the sprint either in Halle (Saale), Germany or remote. We are looking forward to a good time to put up the banner of Zope in the Python land again.

zodb.py3migrate: Migrate an existing ZODB Data.fs to be used with Python 3

At Berlin Strategic sprint 2016 we developed a tool to analyze a ZODB Filestorage to find Python 2 string objects. If they are in an encoding besides ASCII this is preventing using this Filestorage with Python 3 because of decoding errors arising on loading the pickles.
The tool is even able to convert those strings either to unicode by decoding them using a configurable encoding or convert them to zodbpickle.binary so Python 3 will read them as bytes.
There is documentation of the tool and a repository on GitHub where the code lives.

There are still some questions open:

  • Is there already another tool for this analysis/migration?
  • Is there already any practical knowledge migrating Filestorage contents to Python 3?
  • Do you think such a tool is the right approach to achieve such a migration?
  • Is there anyone who wants to try out the tool on a Filestorage of a personal project and share the experiences? (We analyzed two projects where we have access to a Filestorage but we are sure this does not catch all the edge cases.)

Making CI results visible at Home

We are using a Jenkins server to test our projects continuously. To make the results clearly visible, my colleagues installed a Raspberry Pi to display the testing results on a huge LED strip. As you see in the picture below, the project results are displayed on both sides of the whiteboard. The aggregated result is displayed in the big LED tube on the top.

Picture of our Whiteboard with LEDs on either side displaying test results

This setup was built in 2012, but this year we started to work remotely, i.e. from home. Therefore we have no access to informations which are physically displayed at the office. Some month ago, Daniel Havlik mouthed the idea to build an LED strip “to go”. Since the decreased visibility annoyed me, I was a strong advocate of his idea.

Finally, during the Summer Sprint we held in September, we had the opportunity to work on a project of our choice. Daniel Havlik supplied us with an Arduino Uno and a small LED strip, so we started to work on his idea together with a friend of mine, Oliver Zscheyge.

During the first day, we built a simplistic API for the Arduino to set the color of an LED by providing position and color. To test the API we wrote a small Python script that sent random colors to all LEDs.

At the second day of the sprint, Daniel astonished us, since he had put in some extra hours during the night to build a prototype. It had a wooden frame, colored in black and all. Wow! This was definitely the most “visible” result of the Summer Sprint.

Prototype in a wooden case with LEDs and a USB connector

Later that day, we extended the Python script to retrieve the project states from Jenkins and set the LEDs accordingly. We used an INI-file to define which LED represents which project. For fun we also played a short sound when a project was working and broke or the other way around.

Since one of my co-workers wanted to move back to his home town, he got the prototype. To speak truly, I was a little bit jealous about that. But to my surprise, Daniel built another “LED to go” and gave it to me in October, as a birthday present. Since I am a LARPer, I love the shape of it which resembles a sword.

Remote LED with a Sword-like shape

At some days I like to pronounce that “my sword of Jenkins power is glowing green”, which casts a smile on the faces of my co-workers. I think this was a great project with a very useful result.

If you possess some crafting abilities, you can easily make an “LED to go” yourself. We made our code public at bitbucket/remoteled. It contains the versatile API for the Arduino, as well as the Python script which retrieves data from a Jenkins server. Of course you can build additional clients for your personal needs. If you think it’s useful to others as well, send us a pull request. We are looking forward to it!

With these words I am wishing you all a Merry Christmas and a Happy New Year, since I will not write another post in 2014. See you in January!

September, 18th–20th: DevOps Sprint

Since we have a strong history in web development, but also were involved in operating web applications we developed, the DevOps movement hit our nerves.

Under the brand name “Flying Circus” we are establishing a platform respecting the DevOps principles.

A large portion of our day-to-day work is dedicated to DevOps related topics. We like to collaborate by sharing ideas and work on tools we all need to make operations and development of web applications a smooth experience. A guiding question: how can we improve the operability of web applications?

A large field of sprintable topics comes to our mind:

Logging

Enable web application developers to integrate logging mechanisms into their apps easily. By using modern tools like Logstash for collecting and analyzing of the data, operators are able to find causes performance or other problems efficiently.

Live-Debugging and Monitoring

Monitoring is a must when operation software. At least for some people (including ourselves), Nagios is not the best fit for DevOps teams.

Deploying

We always wanted to have reproducable automated deployments. Coming from the Zope world, started with zc.buildout, we developed our own deployment tool batou. More recently upcoming projects, such as ansible, and tools (more or less) bound to cloud services like heroku.

Backup

After using bacula for a while, we started to work on backy, which aims to work directly on volume files of virtual machines.

and more…

Join us to work on these things and help to make DevOps better! The sprint will take place at our office, Forsterstraße 29, Halle (Saale), Germany. On September, 20th we will have a great party in the evening.

If you want to attend, please sign up on http://www.meetup.com/DevOps-Sprint/events/191582682/.

 

Accomodation

For your stay in Halle, we can recommend the following Hotels: “City Hotel am Wasserturm”, “Dorint Hotel Charlottenhof”, “Dormero Hotel Rotes Ross”. For those on budget, there is the youth hostel Halle (http://halle.djh-sachsen-anhalt.de/). Everything is in walking distance from our office.

Run tests using layers with py.test

TL;DR

Long Story

We have many test suites which use test layers (e. g. the ones from plone.testing). We want to use py.test and all its fancy features to have a modern test runner. There was no way to convert such tests partly: either you have to port the whole project or you are stuck with the zope.testrunner.

On our Pyramid-Sprint Godefroid Chapelle, Thomas Lotze and me wrote a package which wraps layers as py.test fixtures. The result is gocept.pytestlayer.

Implementation

For each layer it creates two fixtures: one for the layer setUp/tearDown and one for the testSetUp/testTearDown. The layer fixture is configured for class scope but the plug-in orders the tests and knows about the next test so the layer is only torn down if the next test needs another fixture.

Usage

You only have to add a new section to your package buildout and running the test via

bin/py.test -x

detects the layers and displays the needed setup code. See the PyPI-Page of the package for details.

Future

Maybe it is possible to get rid of the fixture setup code, so running tests using layers gets even easier.

#monitoringlove sprint takeaway

A few weeks ago I co-organised and participated in a #monitoringlove sprint in Berlin.

My personal plan was to play with more modern utilities that can potentially replace our existing Nagios monitoring chain. The result of what I think would be a good setup would probably look like this:

monitoringlove2

Most of those parts already exist. The new thing in there is what I called “riemann-actual” – something that generates new events based on existing events from the index. I call this “higher order” monitoring – in Nagios these would be known as “business processes”.

The word “business processes” is a bit misleading as nothing is really about processes there: it means taking previously taken monitoring data and subsuming it into a more dense expression. Ideally you can recombine any of your metrics to make an overall statement of “everything is good if more than 80% of the appservers are up and we have less than 5% of error response rate and the frontpage is reachable from at least 3 outside systems”.

Data gathering

First, I tried to setup something for data gathering. I already got the recommendation to look at scales for in-app metrics and found it easy to get started. I like the notion that metrics in your app behave a little bit like logging: you don’t care where they go and you expect the user of your system to configure an actual target. The built-in webserver is nice to get started and graphite as a protocol seams fair enough nowadays to forward data.

To gather system-level metrics I guess both collectd and statsd are fine points to start from. I used collectd to begin with as it actually had a riemann output plugin.

Central processing

We want to be able to take all of the data we acquire into account on making decisions quickly. Riemann seems to be the most suitable tool for this task. After playing around for a while trying to implement “business process” monitoring in clojure I found it easier to provide a Python-environment that can talk to riemann and do those decisions. I made this available as “riemann-actual” on bitbucket.

I noticed that this setup would require only a very generic riemann configuration and could perform on a per-customer or per-project basis by just adding more of those loops on top of riemann.

Performance-wise I was extremely happy. I could have a 10Hz monitoring loop resulting in about 1k events per second on my computer. With that resolution all business processes would notice an outage with no visible delay.

A nice feat is that Riemann can generate events when old events reach their TTL. This way you can make sure that you notice when a system you are monitoring “goes dark”.

Also, it seems that Riemann configuration can be unit-tested easily: feed events in, watch the index, or see events coming out. It doesn’t get much simpler than that.

The configurable dashboard in Riemann 0.2 is already very helpful: responsive, flexible, and fast  – until you try to display 10k metrics at once. 😉 It needs a little more finishing but it’s on a good way.

Distributed consumers

My understand of Riemann is that it wants to be a nexus for “central, volatile, shared state”. This means you get a lot of updates going through and that it needs to be good with I/O. OTOH it means that it shouldn’t do much and just make it easy for you to router your data somewhere else.

Actually looking at further consumers didn’t happen as 3 days aren’t that long. 🙂 I see graphite on the horizon (with the setup becoming easier over time) as well as more custom tooling to turn events into notifications, etc.

A look at OpenTSDB seemed promising at first but it turns out to have an even more complex setup requirement than graphite. I got it running but it seemed extremly hard to control, so I dropped it after a few hours.

Overall it seems that since the outcry of #monitoringsucks a lot has happened and I’m faithful that there’s a way out of Nagiosland in the near future.

More notes from our sprint are available at pysprints.de. (Although in German.)

 

August, 15th–17th: Sprinting on Pyramid

After Zope “-the-Framework” reaching the end of its lifecycle during the last few years, we did a bunch of new projects with Pyramid, a nice web framework primarily authored by long-term Zope developer Chris McDonough.

We think it’s about time to give something back to the community, and become more involved in Pyramid development. We therefore happily announce to host a large Pyramid sprint organised in cooperation with the pysprints.de team. You find more information and sprint topics at GitHub.

The sprint starts on Thu, August 15th 10:00h CEST, and ends on Sat, August 17th with a garden party in the evening! Expect BBQ, beer and (most likely) live music!

If you would like to attend, please sign up on lanyrd.

Notizen aus dem Norden

Unsere Herberge auf Rügen 2012
Unsere Herberge auf Rügen 2012

Heute neigt sich unser diesjähriger Sprint dem Ende. Am Dienstag sind wir gen Norden auf die Insel Rügen gefahren, um uns für 3 Tage intensiv darüber zu unterhalten, wie wir in Zukunft Software entwickeln wollen. Der Fokus lag dabei auf der Client-Seite, also dem Teil der Anwendung, der im Browser ausgeführt wird. Hier wollen wir weitere Erfahrung sammeln und Konzepte suchen, wie man wart- und testbaren, strukturierten Code schreibt.

Im Vorfeld haben wir uns auf drei JavaScript Frameworks/Bibliotheken geeinigt, die wir uns dann am Mittwoch in drei Gruppen à zwei Personen intensiv angeschaut haben: Ember, Obviel und Backbone. Neben Dokumentation lesen und Tutorials ausprobieren haben wir auch gegen einen selbst-gestrickten “Spielserver” programmiert, um durch die praktische Erfahrung ein Gefühl für die Konzepte hinter dem Framework zu bekommen und einschätzen zu können, inwieweit sich das Framework in unserem Alltag einsetzen lässt.

Ember hat uns für den Einsatz bei “Grüne Wiese”-Projekten begeistert. Durch die strikte Trennung von Model, View und Controller (MVC) schreibt man nicht nur automatisch saubereren, sondern auch testbaren Code. Das Objekt-Modell bringt automatisierte, bidirektionale Aktualisierung zwischen Model und View mit, und für die Speicherung auf dem Server gibt es ein kleines, gut integriertes REST-Plugin. Der Aufwand, eine funktionierende CRUD-Anwendung, die Änderungen automatisch an den Server propagiert, zu entwickeln, ist mit Ember erstaunlich gering. Auch wie Ember es erlaubt, seine Klassen zu beerben, ist uns positiv aufgefallen. Leider haben wir es nicht geschafft, zwei Ember-Anwendungen auf einer Seite zu laden. Auch ist die Dokumentation noch recht dünn, was daran liegen dürfte, dass Ember noch ein junges Produkt ist.

Hilfreiche Links zu Ember:

Obviel ist streng genommen unser eigenes “Kind”, denn erste Ideen dazu entstanden im Jahre 2008 bei gocept. Obviel ist ein Client-seitiges Web-Framework für jQuery mit einem einfachen, aber genialen Prinzip: für ein JavaScript-Objekt rendert man einen View in einem Element. Dreh- und Angelpunkt bei Obviel sind die Interfaces, die definieren, wie ein View Objekte, die das Interface implementieren, rendert. Entwickler, die mit der Komponenten-Architektur von Zope vertraut sind, werden keine Probleme haben, Obviel zu verstehen. Weiterhin lässt sich Obviel problemlos in bestehende Projekte integrieren, es enthält Unterstützung für Internationalisierung (i18n), Forms mit Validierung, verschiedene Template-Engines, Subviews, …

Hilfreiche Links:

Backbone unterstützt den Entwickler bei der Strukturierung seiner Webanwendungen durch in Code gegossene Konventionen. Es wirkt sehr fokussiert, denn es tut nicht wirklich viel, aber das Wenige tut es sehr sinnvoll. Backbone ist, im Gegensatz zu Ember, sehr gut dokumentiert, löst aber im Moment keines unserer Probleme. Da Backbone insbesondere keine Unterstützung für data binding bietet, haben wir noch Knockout und Rivets ins Spiel gebracht. Für ersteres steht mit Knockback der „Glue-Code“, um die Frameworks miteinander zu verheiraten, schon zur Verfügung, für letzteres mussten wir ihn uns aus der Dokumentation und dem Issue-Tracker des Projekts zusammensuchen.

Weiterführende Links:

Ein Fazit zu ziehen fällt uns nicht leicht: Ember behalten wir für das nächste “Grüne Wiese”-Projekt auf jeden Fall im Auge. Da uns Backbone und Knockout/Rivets nicht überzeugen konnten, haben wir uns noch spontan AngularJS angeschaut – aber auch hier ist im Moment die Ernüchterung groß: Persistieren von Daten ist ein ungelöstes Problem, die Watcher auf dem Model (data binding) feuern nur unregelmäßig oder viel zu oft. Nach längerem Probieren haben wir herausgefunden, dass wir eine eigene Direktive schreiben müssen, die sich um die Persistierung kümmert.

Bleibt also Obviel, das wir an der einen oder anderen Stelle in unseren Projekten schon verwenden. Es wird wohl auch in Zukunft unser Werkzeug, um bestehenden JavaScript-Code zu verbessern oder neue Frontend-Einheiten zu bauen.

Das gocept-Team