No luck with glusterfs

Recently, we’ve been experimenting with glusterfs as an alternative network storage backing our VM hosting. It looked like a very promising candidate to replace our current iSCSI stack: scale-out with decent performance, mostly self-configuring, self-replicating, self-healing. And all of this out-of-the-box without complex setup. In contrast, the conventional architecture with a complex layering of iSCSI targets, DRBD, and Linux-HA glued together with a pack of shell scripts looks rather 90’s.

We played with glusterfs for a while. Setting up and configuring the software went quite smooth compared to the traditional stuff. But after some stress testing in a replicated scenario, we found severe problems.

Synchronisation

On the storage, the virtual machines represent themselves basically as one big image file. This image can become several hundreds of Gigabytes big. This is OK as long as the replicated file servers are in sync. But once one goes offline and online again, the versions of the image may differ and the self-healing algorithm is triggered. Due to glusterfs’ architecture, this happens  entirely on the filesystem client (i.e., the KVM host). After re-connecting a file server, all VM I/O is to be paused until self-healing is complete. The live VM is stuck for some amount of time between several seconds and more than a minute. A considerable portion of our hosting cluster could freeze for minutes. This is cleary unacceptable. Re-connecting a previously disconnected file server would be a risky operation: quite the opposite of what replication is good for.

No global state

Another feature of glusterfs is that replication is handled entirely on the filesystem client and not on the server. This leads to an orthogonal and modular approach which has a lot of advantages. But it makes it hard to determine when a file server can be disconnected safely: Given that self-healing takes a considerable amount of time, we cannot be sure if there is still some self-heal operation in progress. But disconnecting a replicated file server which had the newer copy of a VM image before the other file server has caught up would render the VM unusable. Unfortunately, there seems to be no easy way to query a glusterfs file server for active self-healing operations. This makes disconnecting a file server a risky operation, too.

Good for its intended use

In summary, we learned that glusterfs’ architecture is a good fit for the use case it has originally been designed: a NFS replacement with lots of small files. But for our scenario where continuously running processes need to access a few large image files uninterruptedly, glusterfs seems not to be the best fit.

So we will stick to the good ol’ iSCSI stack for now. Perhaps Ceph or Sheepdog will become viable alternatives in the future once they stabilise.

rrdtool restore and merge from backup

We recently had an issue with our backup server which was also running Nagios including pnp4nagios to gather performance data.

We quickly started to deploy a new Nagios server which started gathering statistics again right away.

After pulling the historical RRD databases from the backup we discovered no easy way to integrate the both datasets. After fiddling with some tools we extended an existing script that can be used to integrate different RRD sources into a single file to match our use case.

The resulting script simply replaces all “null” data rows in the new database that the old database has data available for. A second script provides the ability to merge whole directory trees of RRDs.

The script can be found in the rrdmerge bitbucket repository.

How-To: Undo a transaction with the ZODB

Suppose you’ve written a script to “fix something real quick” and unleashed it upon your live database. Five minutes later, you discover your script had a bug, and now you’ve wrecked quite a bit of production data. Ouch.

You might be lucky, though, since the ZODB offers transaction-level undo. This comes with a lot of caveats, though, the biggest being that if something else was changed in the meantime that causes the undo to conflict, it won’t work. (Before transaction X, some value was A which X changed to B, but later something changes it to C. If I now want to undo transaction X to get back to A, it will conflict. Catalogs and other shared state are prime candidates).

But you still might be lucky and there won’t be a conflict. So, how do you undo a transaction? First, you need to find the transaction. In my case, I knew an object that had been changed by my script. So I asked the ZODB for the history of that object, i.e. the last transaction(s) that changed it:

>>> db = root._p_jar.db()
>>> hist = db.history(my_changed_object._p_oid)
>>> hist
[{'tid': '\x03...', 'size': 123, 'user_name': '', 'description': '', 'time': 1304493667.320477}]

Now I have the offending transaction’s ID. However, the undo() API does not work with transaction ids but needs a special (storage-specific) identifier. And, since as far as I can tell there is no way to map a transaction id to an “undo id”, I had to make to by matching the time stamp:

>>> info = db.undoInfo(specification=dict(time=hist['time']))
>>> info
[{'id': 'A44XmR876bs=', 'time': 1304493667.320477, 'user_name': '', 'description': '', 'size': 315893}]

Finally, call undo and hope you don’t get a conflict upon committing:

>>> db.undo(info['id'])
>>> import transaction
>>> transaction.commit()

Get a zc.sourcefactory to implement an interface

zc.sourcefactory is very handy to easily create a source (zope.schema.interfaces.IIterableSource to be precise) with corresponding titles and tokens for its contents. Every now and then a source requires an explicit interface. For zc.sourcefactory the following code snippet helps:

class IMySource(zope.schema.interfaces.IIterableSource):
    """my source"""

class MySource(
    zc.sourcefactory.contextual.BasicContextualSourceFactory):
    """The source factory."""

    class source_class(
        zc.sourcefactory.source.FactoredContextualSource):
        """This class is being instanciated by the factory.

        It *must* be called source_class.
        """
        zope.interface.implements(IMySource)

    def getValues(self, context):
        …

Of course it is also possible to declare the source_class separately from the source factory and reference it. But since its sole purpose is to hold an implements declaration, I’m fine with defining it inline.

Testing pyramid applications with zope.testbrowser

The Pyramid documentation recommends using WebTest for functional tests, but coming from the Zope world, zope.testbrowser is the more familiar tool for this job (and it seems a little more high-level than WebTest at first glance).

With the 3.11 release,  zope.testbrowser gained out-of-the-box support for talking to WSGI applications (courtesy of wsgi_intercept), so herewith some corresponding example code to show the basic usage and setup. (This assumes you are using zope.testrunner or something else that supports its layers).

import unittest
import zope.testbrowser.wsgi

class MyAppLayer(zope.testbrowser.wsgi.Layer):

    def make_wsgi_app(self):
        from myapp import main
        return main({})

MY_APP_LAYER = MyAppLayer()

class FunctionalTests(unittest.TestCase):

    layer = MY_APP_LAYER

    def test_root(self):
        browser = zope.testbrowser.wsgi.Browser()
        browser.open('http://localhost/')
        self.failUnless('Pyramid' in browser.contents)

Adding round corners to boxes using CSS and JavaScript

Curvycorners has a way to make gorgeous roundings to your boxes without having to split each box into pieces of background images. You simply need to include one JavaScript file and add two lines of CSS code to your box (or to the box’s class definition).

First, include the curvycorners javascript file in the head part of your webpage.

Then, add the following lines to the box that should get round corners. Here you can also customize the radius:

-moz-border-radius: 3px;-webkit-border-radius: 3px; 

That’s all, the result looks like this:

A box with curvy corners.

Localized dates with Zope 2 and Zope 3

Imagine, you want to display a date in a nicer way than just 2009-11-17. It’s not that difficult to localize it into e.g. German and output 17.11.2009. But what, if you need to display Mittwoch, 11.November 2009 instead?

Calling strftime(‘%A, %d.%B %Y’) on your date object would format the date with month- and week names, but in most cases display the names in English and not in German, as we @gocept need it.

Locales provide you with the nessecary functionality to format your datetime object in the users local format and language. You can choose from different templates (parameter length) or even create your own templates.

The example defines a method formatDate which takes a datetime object. It retrieves the users locale settings from the request, chooses a format template and then “mixes” everything together :-).

So, to get something like “Wednesday, 17.November 2009”, you will need to submit the length parameter full as shown in the above example:

class MyView(object):
def formatDate(self, datetime_date):
if not self.request._locale:
# zope2 doesn't initialize the locale itself
self.request.setupLocale()
formatter = self.request._locale.dates.getFormatter(
'date',  length='full')
return formatter.format(datetime_date)