Zope in the Python 3 wonderland

A little tale

Once upon the time there was the big mighty Zope II. It was one of the leaders in the Python land. It had mighty features like TTW (trough the web) development and its own object oriented database. Many people liked Zope II and trusted it to be the basis for their personal work and the work of their companies.

After some years Zope II got a son named Zope III. It could have been even mightier and more trustworthy than Zope II. But some people argued that Zope III is too complex and too hard to understand and way too different from Zope II which they knew very well.

New, faster and easier to understand frameworks arose beside Zope II and Zope III and the people began to like the new ones more. After many years of power, Zope became weak. Its name got hated by many people in its country. It even got nearly forgotten.

But there are still the people out in the far away kingdoms who built a complete ecosystem on the shoulders of Zope. They cannot completely change the basis of their work within months or even years. They deserve a perspective even for the Python 3 wonderland which is no longer a vague dream but already a future with wide open doors.

Back in reality

We are here to help Zope to find a new home in the Python 3 land and live there happily a great many years. Therefore we scheduled a Sprint named Zope Resurrection Sprint where we will discuss and code on Zope to get it more Python 3 ready.

The big blocker is currently RestrictedPython which relies on modules which are no longer in the Python 3 standard library. But RestrictedPython is an important part of the TTW security concept, so it has to be rewritten from scratch. Folks who think they do not need TTW in their projects can work on a Zope where RestrictedPython is optional.

Besides the Python 3 migration of the code the contents of the application databases using Zope need a conversion, too. On a previous sprint in Berlin, we wrote a possible solution: zodb.py3migrate which needs some more tests with real databases.

Furthermore, there is Zope IV, the successor of Zope II which needs some love to become strong and ready for its adventurous journey as the basis of Plone VI and others.

It is still possible to join the sprint either in Halle (Saale), Germany or remote. We are looking forward to a good time to put up the banner of Zope in the Python land again.

zodb.py3migrate: Migrate an existing ZODB Data.fs to be used with Python 3

At Berlin Strategic sprint 2016 we developed a tool to analyze a ZODB Filestorage to find Python 2 string objects. If they are in an encoding besides ASCII this is preventing using this Filestorage with Python 3 because of decoding errors arising on loading the pickles.
The tool is even able to convert those strings either to unicode by decoding them using a configurable encoding or convert them to zodbpickle.binary so Python 3 will read them as bytes.
There is documentation of the tool and a repository on GitHub where the code lives.

There are still some questions open:

  • Is there already another tool for this analysis/migration?
  • Is there already any practical knowledge migrating Filestorage contents to Python 3?
  • Do you think such a tool is the right approach to achieve such a migration?
  • Is there anyone who wants to try out the tool on a Filestorage of a personal project and share the experiences? (We analyzed two projects where we have access to a Filestorage but we are sure this does not catch all the edge cases.)

Manage JavaScript dependencies with BowerStatic

Last month I explained how to use Fanstatic to manage JS dependencies. Since we were more and more displeased by using Fanstatic, we recently switched to BowerStatic, the new kid on the block. Since the setup is a bit more complicated and you need more tools to have the same features as you had with Fanstatic, I will describe how to set up the full toolchain. After that, I will shortly summarize the benefits and drawbacks of BowerStatic, so you can decide for yourself if you prefer BowerStatic above Fanstatic or the other way around.

Of course the choices made in the toolchain of BowerStatic are strongly linked to other tools we use. Since we set up large projects using Buildout, we like to use Buildout recipes to solve a problem. Therefore this blogpost is most interesting when you are using Buildout yourself.

How it works

BowerStatic was created by Martijn Faassen, who was a core developer of Fanstatic. We think of BowerStatic as a reimagined version of Fanstatic, avoiding disadvantageos decisions made in the past. One problem of Fanstatic was, that it tried to do too many things: Fetching the files, delivering them to the client and bundling them into one large file.

Therefore BowerStatic has a narrow focus: It only helps to manage which libraries should be delivered to the client. It does not care how the files were retrieved or whether you want to have a bundled resource. To automate these steps as well, you must look for additional tools.

We use the following toolchain:

  • a Buildout recipe for Bower to download all external libraries
  • BowerStatic to deliver JS files to the client
  • a custom Buildout recipe to bundle all files of the project into one large file

Since you need all of these tools to have the same features as Fanstatic, I will give a short description, how to set things up.

Bower

Bower is a small tool to download libraries and their dependencies by using a simple description of the library in JSON, which is named bower.json. The description contains a version string, the path to contained files, a list of libraries it depends on and a lot more metadata like author, homepage, keywords and so on. For example:

{
  "name": "bootstrap",
  "version": "3.3.2",
  "main": [
    "less/bootstrap.less",
    "dist/css/bootstrap.css",
    "dist/js/bootstrap.js",
    "dist/fonts/glyphicons-halflings-regular.eot",
    "dist/fonts/glyphicons-halflings-regular.svg",
    "dist/fonts/glyphicons-halflings-regular.ttf",
    "dist/fonts/glyphicons-halflings-regular.woff"
  ],
  "dependencies": {
    "jquery": ">= 1.9.1"
  }
}

The great thing about Bower is, that you just point to a github repository in case this certain library is not registered on the Bower server. In this case it will check out the trunk of the repository and creates the bower.json all by itself using the commit hash as it’s version. Of course it cannot auto-detect dependencies, which are normally declared in the bower.json.

It’s even better if the repository contains a bower.json file. In this case Bower will only load the files mentioned there and can retrieve dependencies. Therefore it makes no difference if you point Bower to a registered package, e.g. jquery-ui, or if you point it to the github repository of this project, e.g. github.com/jquery/jquery-ui. Of course the version will be different (stable release vs trunk).

To download external dependencies via Bower, we added bowerrecipe to our Buildout configuration. This way we can just list the libraries and their version in the recipe configuration. Bower will automatically download those libraries, as well as their dependencies. The following example will download Bootstrap and jQuery, since Bootstrap depends on jQuery.

[bower]
recipe = bowerrecipe
packages =
    bootstrap#3.3.x
executable = ${buildout:bin-directory}/bower

By default the libraries will be stored inside parts/bower/downloads. The destination can be changed inside the recipe configuration.

BowerStatic

BowerStatic uses libraries downloaded via Bower to resolve JS dependencies and delivers all required files to the client. To do so, it must be registered as a WSGI middleware. Compared to Fanstatic, the setup of BowerStatic is a little bit more complicated:

  1. Create an instance of the Bower class
  2. Create a external component collection, which contains all external libraries loaded into the directory parts/bower
  3. Create a local component collection, which will contain your self-written JS files
  4. Set the external collection as the fallback of your local collection, i.e. JS libraries not found in the local collection will be looked up in the external collection
  5. Add your self-written files to the local collection, by reading a bower.json at given path

This may sound like voodoo to you. The documentation of BowerStatic contains some insightful statements why the setup is so complicated. Written in code the setup will look like this:

import bowerstatic
import os
import pkg_resources

bower = bowerstatic.Bower()
external_components = bower.components(
    name='components', path='parts/bower/downloads')
local_components = bower.local_components(
    name='local', component_collection=external_components)

local_components.component(path=pkg_resources.resource_filename(
    'my.cool.package', 'resources'), version=None)

Let’s assume in my.cool.package.resources is a bower.json like:

{
  "name": "custom",
  "version": "0.1",
  "main": [
    "custom.js",
    "custom.css",
  ],
  "dependencies": {
    "bootstrap": "*"
  }
}

Then you have the JS library called custom in your local component collection. After handling this setup, you can use BowerStatic similar to Fanstatic, by including JS files inside any view. However, you always need the component collection to do so. local_components.include(self.request.environ, 'custom') will load custom.js, custom.css, the bootstrap library and the jQuery library. You could also write local_components.include(self.request.environ, 'jquery'), which would only load jQuery.

Bundling

Since we were used to Fanstatics automatism to bundle and minify all resources, we wanted to have something similar with BowerStatic. It is not sure if BowerStatic will ever contain bundling mechanisms, since this may be out of the narrow scope it wants to preserve.

Therefore we build our own Buildout recipe, gocept.recipe.bowerstaticbundler. This recipe will create another Bower library in parts/bower, which contains a minified JS file, a minified CSS file and assets like images and fonts. By using the dependency mechanism of BowerStatic, all JS files are bundled in the right order.

To use the bundled library in production and the separate files in development mode, we build a custom include with an environment variable switch:

def bower_include(environ, name):
    if not os.environ.get('BOWERSTATIC_DEBUG'):
        name = 'bowerstatic_bundle_' + name.replace('.', '_')
    include = local_components.includer(environ)
    include(name)

Of course you must use this include method instead of using the BowerStatic component directly, i.e. replace alo occurences of local_components.include(environ, name) with bower_include(environ, name).

This solution is good enough for ourselves, but may not be ready for your purposes. We have build it with a certain project in mind, so you might come across bugs when using it with different JS libraries. Since we will soon use it in production, we will happily fix bugs you report and merge tested improvements. You can find the code and issue tracker on bitbucket.

Benefits

The combination of Bower, BowerStatic and a bundling mechanism has the same benefits as Fanstatic, i.e. that you no longer need to copy JS files into your project and link them statically, as well as the benefits of bundled resources to speed up the initial page load.

The biggest contrast of these solutions is how external libraries have to be prepared: With Fanstatic you had to build an integration package for each library and release a version on PyPI for each version of the library. With BowerStatic it’s enough to have a current bower.json inside the library. So both need some kind of work and maintenance.

However, with Fanstatic the assumption was that someone else already built the integration package for you, so that you had to do nothing. As described in the Fanstatic article from last month, this assumption is no longer valid and many JS libraries are not supported or out of date. Updating a foreign integration package consumes time and may not be possible at all, in case the user is no longer active.

With BowerStatic, you can simply point to a git repository and select the version by using a certain commit or tag. As mentioned before, Bower will download the trunk of the repository and generates a simple bower.json automatically. Therefore every JS library that is hosted on github can be used with BowerStatic without additional work. This means no integration packages anymore, yay!

The downside is that you must declare dependencies of those libraries as your dependencies, since Bower cannot detect dependencies when it automatically creates a bower.json from the trunk of the repository. However, if there is a bower.json inside the repository, Bower will use it and to resolve dependencies.

The possibility of Bower to download a github repository comes with the benefit that it is really easy to try out different libraries for the same task. With Fanstatic you had to build an integration package for every experiment or copy the files into your resources. Now you can just point to the github repository and give it a try. Fanstatic always felt like a big hindrance when trying out new libraries.

Drawbacks

In my opinion the biggest drawback of BowerStatic is the long toolchain. You have to understand how Bower works, how BowerStatic uses Bower and how to set it up. Building a custom recipe for bundling was a bit time consuming as well.

After about 2 weeks we had aggregated enough wisdom to use the toolchain smoothly. We think of it as an investment in the future, since we were so annoyed by Fanstatic, building one integration package after the other.

Another big downside is, that there is no obvious solution to bundling when using BowerStatic. Of course you could bundle the resources using Grunt or any other approach of your choice. But we wanted a solution that fits into our toolchain, i.e. a solution that works with Python and integrates into Buildout. We did not know of any solutions that matched this criteria, therefore we created the Buildout recipe ourselves. However, building the recipe consumed some time and it is nowhere near “complete”. I hope that there will be a bundling solution that integrates nicely with BowerStatic in the near future. Till then, the Buildout recipe is good enough.

Conclusion

Wow, that was a long article. I hope it was interesting for you to read about BowerStatic. Chances are that you did not know about it before. Setting it up did take some time, but maybe the code snippets above help you to speed things up.

All in all I am really glad that we switched to BowerStatic. After the initial learning investment, you know almost everything that you need to know for the future. Testing brand new JavaScript libraries is a breeze. Even using an unknown library takes no additional effort. Sounds like a win to me.

Manage JavaScript dependencies with Fanstatic

Until the beginning of this year, we were using Fanstatic to manage dependencies to external JavaScript libraries. In case you are not familiar with Fanstatic, here is a short overview. I will discuss benefits and drawbacks later on.

How it works

Imagine you want to use jQuery in one of your projects. Therefore you build an integration package, say js.jquery, which contains the jquery.js file. You also add a resource.py which would look like this:

from fanstatic import Library, Resource

library = Library('jquery', 'resources')
jquery = Resource(library, 'jquery.js', minified='jquery.min.js')

To finish up, you state the version of the downloaded jquery.js in the setup.py and upload the package to PyPI.

After wrapping your app with Fanstatic, you can declare JavaScript dependencies by listing the integration package inside your setup.py. To use them, you write js.jquery.need() inside any view. Of course you can also add a resource.py and declare a local resource with dependencies:

import fanstatic
import js.jquery

library = fanstatic.Library('custom', 'resources')
my_awesome_js_code = fanstatic.Resource(
    library, 'js/my_awesome_js_code.js',
    minified='js/my_awesome_js_code.min.js',
    depends=[js.jquery.jquery])

Every time you call my_awesome_js_code.need() inside a view, it will also load jQuery.

Benefits

With Fanstatic you no longer need to copy the JavaScript files inside your project and link them statically, you just declare them as a dependency to your own JavaScript code and call my_awesome_js_code.need() inside a view.

The preparation described above can be tedious, but the assumption is, that someone else has built the integration package before, so you can reuse it. Installing a dependency therefore is as easy as adding a line to your setup.py and declare it as a dependency of one of your resources.

Fanstatic makes it a lot easier to update external libraries, since you only need to increase the version number in your setup.py. You can also make sure to use the same version across many projects, e.g. by sharing some buildout configuration across projects.

And besides many small nice features, Fanstatic can also create bundles, i.e. it will merge my_awesome_js_code and all of its dependencies into one big file and deliver it to the client, rather delivering all files separately. This usually makes the initial page load much faster.

Drawbacks

Despite all the benefits Fanstatic offers, we were getting more and more frustrated with it. The main reason is, that we feel a decline in the activity of the Fanstatic user group. With fewer people using Fanstatic, we often have to build the integration packages ourselves. This eliminates one of the main benefits, i.e. that we could reuse the integration packages of others.

For example, in a big project we did last year, we used integration packages for js.jquery, js.classy, js.chosen and many more. Those are pretty popular packages and the integration worked out of the box.

However, even those packages have issues: The integration package for jQuery does not offer version 2.0 or higher and Chosen is only available in a single (outdated) version. We often stumbled upon similar issues with other libraries, e.g. the integration package for js.modernizr is more than 2 years behind.

In addition, some newly developed libraries may not be available at all. Of course we could write the integration packages for those or update the jQuery integration package. But this would mean to find out where the code for an integration package is hosted, write a pull request and wait for a merge, as well as a new release on PyPI. In case the owner is not active anymore, you are out of luck and must add a duplicate package to PyPI. Tedious and unsatisfactory.

This is why we started to look for an alternative to Fanstatic. And maybe we have found it, but this is a story for another day, where I will talk about our experiences with Bower and Bowerstatic.

Making CI results visible at Home

We are using a Jenkins server to test our projects continuously. To make the results clearly visible, my colleagues installed a Raspberry Pi to display the testing results on a huge LED strip. As you see in the picture below, the project results are displayed on both sides of the whiteboard. The aggregated result is displayed in the big LED tube on the top.

Picture of our Whiteboard with LEDs on either side displaying test results

This setup was built in 2012, but this year we started to work remotely, i.e. from home. Therefore we have no access to informations which are physically displayed at the office. Some month ago, Daniel Havlik mouthed the idea to build an LED strip “to go”. Since the decreased visibility annoyed me, I was a strong advocate of his idea.

Finally, during the Summer Sprint we held in September, we had the opportunity to work on a project of our choice. Daniel Havlik supplied us with an Arduino Uno and a small LED strip, so we started to work on his idea together with a friend of mine, Oliver Zscheyge.

During the first day, we built a simplistic API for the Arduino to set the color of an LED by providing position and color. To test the API we wrote a small Python script that sent random colors to all LEDs.

At the second day of the sprint, Daniel astonished us, since he had put in some extra hours during the night to build a prototype. It had a wooden frame, colored in black and all. Wow! This was definitely the most “visible” result of the Summer Sprint.

Prototype in a wooden case with LEDs and a USB connector

Later that day, we extended the Python script to retrieve the project states from Jenkins and set the LEDs accordingly. We used an INI-file to define which LED represents which project. For fun we also played a short sound when a project was working and broke or the other way around.

Since one of my co-workers wanted to move back to his home town, he got the prototype. To speak truly, I was a little bit jealous about that. But to my surprise, Daniel built another “LED to go” and gave it to me in October, as a birthday present. Since I am a LARPer, I love the shape of it which resembles a sword.

Remote LED with a Sword-like shape

At some days I like to pronounce that “my sword of Jenkins power is glowing green”, which casts a smile on the faces of my co-workers. I think this was a great project with a very useful result.

If you possess some crafting abilities, you can easily make an “LED to go” yourself. We made our code public at bitbucket/remoteled. It contains the versatile API for the Arduino, as well as the Python script which retrieves data from a Jenkins server. Of course you can build additional clients for your personal needs. If you think it’s useful to others as well, send us a pull request. We are looking forward to it!

With these words I am wishing you all a Merry Christmas and a Happy New Year, since I will not write another post in 2014. See you in January!

Florian Pilz

Today’s blogpost is all about me. I joined gocept about a year ago, therefore I want to tell you something about the past – and something about the future.

The Past

Florian Pilz

Before joining the gocept crew, I studied computing at the HTWK in Leipzig for 6 years. Since sitting in a german university was not enough for me, I reached out to study in the UK for the best time of a year. This way I came to the University of Bolton near Manchester, where – most importantly – I learned to speak English fluently.

During my time in the UK I started to look into Ruby on Rails and earned my first salaries. This was the point were I switched my Linux PC for a Mac. Since I already was comfortable with Windows and Linux I wanted to try out the last big operation system out there. But as many before me, I liked the Mac so much that I still use it as my development machine.

Back in Germany I continued to earn some money on the side working with Ruby on Rails. I really enjoyed using Ruby and I think in terms of readability it’s ahead of most other programming languages by several magnitudes. But Ruby also has its shortcomings, which I did not recognize until I switched to Python as my main programming language here at gocept.

I also did some very interesting projects during my studies, here are just three of them:

  • An OCR software which recognizes mathematical symbols. The idea was to integrate it into the Leibniz-project to make it possible for blind people to read mathematical school books.
  • An iPhone app that helps developers and freelancers to get better at estimating how long it takes to implement a feature or project.
  • A scientific article about software estimation that was actually published in print!

After I finished my master thesis, I had the opportunity to go on a 2-month vacation with my wife. So we took two backpacks and travelled around the entire US. A memory I will feast on my entire life. You can get some impressions on my flickr fotostream.

The Future

During the last year at gocept, I dove into the code base of two of our biggest clients – DGB and Ver.di – and helped improving their CMS. Currently I am getting in touch with the code base of Zeit Online, since a colleague will move to the upper floor, where our admins are situated.

But most importantly, you will see me more often around here, since I will be writing about things we do and learn here at gocept during the following months. I already have some topics on my desk which are worth a blogpost. For example, why hg rebase can be harmful, why testing AJAX using Jasmine is a pain, the benefits of a headless browser like PhantomJS and many more.

Next up will be a post about something we did at the summer sprint, which includes an Arduino, an LED strip and a failing test server.

Sommerfest bei gocept – Samstag 20. September 2014 ab 16 Uhr

english version below

Kein Sommer geht zu Ende ohne ein Fest in unserem tollen Garten. Wie in den letzten Jahren laden wir unsere Familien, Freunde und Mitstreiter ein. Für leckeres Essen, kühle Getränke und Unterhaltung ist gesorgt – kommt und feiert mit uns – wir bitten um Voranmeldung.

WANN: Samstag, 20. September ab 16 Uhr

WO: Forsterstr. 29, 06112 Halle (Saale)

KANN es nicht erwarten: mail@gocept.com

Übrigens, auch dieses Jahr bildet die Party einen Schlusspunkt zu unserem DevOps Sprint. Hier könnt ihr mehr darüber lesen: gocept DevOps Sprint

 

gocept summer party– Saturday, September 20th  4 pm

No summer ending without a party at the gocept garden. Like in previous years we invite our families, friends, and “brothers-in-arms”. As usual we serve delicious food, chilled drinks, and amusement – so come and join us – R.s.v.p.

WHEN: Saturday, September 20th from 4 pm

WHERE: Forsterstr. 29, 06112 Halle (Saale)

CAN’T wait for it: mail@gocept.com

Incidentally, the party is also the closing event for our DevOps sprint.

September, 18th–20th: DevOps Sprint

Since we have a strong history in web development, but also were involved in operating web applications we developed, the DevOps movement hit our nerves.

Under the brand name “Flying Circus” we are establishing a platform respecting the DevOps principles.

A large portion of our day-to-day work is dedicated to DevOps related topics. We like to collaborate by sharing ideas and work on tools we all need to make operations and development of web applications a smooth experience. A guiding question: how can we improve the operability of web applications?

A large field of sprintable topics comes to our mind:

Logging

Enable web application developers to integrate logging mechanisms into their apps easily. By using modern tools like Logstash for collecting and analyzing of the data, operators are able to find causes performance or other problems efficiently.

Live-Debugging and Monitoring

Monitoring is a must when operation software. At least for some people (including ourselves), Nagios is not the best fit for DevOps teams.

Deploying

We always wanted to have reproducable automated deployments. Coming from the Zope world, started with zc.buildout, we developed our own deployment tool batou. More recently upcoming projects, such as ansible, and tools (more or less) bound to cloud services like heroku.

Backup

After using bacula for a while, we started to work on backy, which aims to work directly on volume files of virtual machines.

and more…

Join us to work on these things and help to make DevOps better! The sprint will take place at our office, Forsterstraße 29, Halle (Saale), Germany. On September, 20th we will have a great party in the evening.

If you want to attend, please sign up on http://www.meetup.com/DevOps-Sprint/events/191582682/.

 

Accomodation

For your stay in Halle, we can recommend the following Hotels: “City Hotel am Wasserturm”, “Dorint Hotel Charlottenhof”, “Dormero Hotel Rotes Ross”. For those on budget, there is the youth hostel Halle (http://halle.djh-sachsen-anhalt.de/). Everything is in walking distance from our office.

Flying Circus at EuroPython 2014

If you’re attending EP14, be sure to visit our Flying Circus booth at BCC level A! We’re here to discuss web operations. Managed hosting is only as good as the people behind it. So just walk over, test us, ask any question related to web operations! Additionally, we have some demo VMs readily available so you can get hands-on experience with a walk-through from our developers.

Flying Circus boot at EP14

Follow up actions after the filesystem corruption incident

On 2014-06-07, the Flying Circus experienced a quite unfortunate filesystem corruption incident. Most of the VMs have been cleaned up since then, but a few defective files are still around. In the following article, I’ll provide some background information on what types of corruption we saw, what you (as our customer) can expect from platform management to rectify the situation, and what everyone can do to check his/her own applications.

Observed types of filesystem corruption

The incident resulted in lost updates on the block layer. This means that some filesystem blocks were reverted to an older state. Depending on what kind of information has been saved in the affected blocks, this may lead to different effects:

  • files show old content, as a file update got lost;
  • files show random content, as updates to the file’s extent list got lost;
  • files have disappeared completely, as updates to their containing directory got lost.

On most VMs which experienced filesystem corruption, filesystem metadata has been rendered invalid as well. We were able to identify these VMs quickly and contacted the affected customers immediately. However, there are still some cases left where filesystem metadata has not been affected (so the automated checks did not find anything), but file contents has been affected. Generally, files that have been updated in the time range between 2014-06-02 and 2014-06-07 or live in directories that saw changes during that time are at risk.

These cases of corruption are impossible to detect via filesystem checks. To make sure that all VMs are in a reasonably good state, twofold action is necessary: First, we will check the OS and all managed components as part of our platform management. Second, we ask you to take a look at your applications to uncover previously hidden cases of filesystem corruption.

Platform-wide checks

After taking short-term action to ensure that we will not run into a similar problem again, we are currently in the process of performing a deep scan of all installed OS files and managed components. In particular, we are going to:

  • perform a consistency check on all files installed from OS packages;
  • perform integrity checks on all managed databases (PostgreSQL, LDAP, …);
  • reboot all VMs to ensure that there is no stale cached content.

Found inconsistencies will be repaired automatically if possible (e.g., OS files). As far as application data is concerned (e.g., databases), we will contact you to work out available options to restore consistency. VM reboots will take place during announced maintenance periods as usual.

Application-specific checks

It is not possible to perform an automated deep check of project files, as we do for OS files. Too much context knowledge is necessary to judge on what is ok and what looks suspicious. So we ask you to throw a critical glance at your applications yourself. Our experience so far shows that signs of filesystem corruption reveal themselves quite fast when one starts to look for the right signs.

Two areas that need to be checked are static project files, like application software and configuration, and application-specific databases.

Checking application installations

Our first and most important recommendation is to restart all applications and check for obvious signs of trouble. This is both easy and points to most problems right away.

Additionally, the applications’ log files should be inspected. Filesystem corruption causes sometimes “illogical” errors to show up in the log files. We recommend to look through the log files for unusual error messages.

If corrupted installation or configuration files are found, the best way out is usually to re-deploy affected applications. This is easy if the deployment is controlled via an automated tool like batou or zc.buildout. Restoring installation files from backup is also an option.

Checking application data stores

Some applications bring in their own data store, for example ZODB or SOLR. Procedures depend on the specific software, but we can give some general suggestions here:

  • Some data stores have their own integrity checking or even repair tools on board. For example, ZODB complains about inconsistencies during packing.
  • Some data stores have means to dump their contents to an external file. During dumping, all pieces of data will be traversed and inconsistencies are likely to be discovered.
  • Some data stores can easily be rebuilt from scratch, for example caches, indexes, or session stores.

Please contact our support if you discover inconsistencies and need help to recover.

Summary

We are sorry for all the trouble the filesystem corruption incident has caused. We care about customer data and will do our best to get VMs as clean as possible. With the guidelines mentioned above, it should be possible to uncover a good portion of the corrupted files that have not been identified yet.