rrdtool restore and merge from backup

We recently had an issue with our backup server which was also running Nagios including pnp4nagios to gather performance data.

We quickly started to deploy a new Nagios server which started gathering statistics again right away.

After pulling the historical RRD databases from the backup we discovered no easy way to integrate the both datasets. After fiddling with some tools we extended an existing script that can be used to integrate different RRD sources into a single file to match our use case.

The resulting script simply replaces all “null” data rows in the new database that the old database has data available for. A second script provides the ability to merge whole directory trees of RRDs.

The script can be found in the rrdmerge bitbucket repository.

How-To: Undo a transaction with the ZODB

Suppose you’ve written a script to “fix something real quick” and unleashed it upon your live database. Five minutes later, you discover your script had a bug, and now you’ve wrecked quite a bit of production data. Ouch.

You might be lucky, though, since the ZODB offers transaction-level undo. This comes with a lot of caveats, though, the biggest being that if something else was changed in the meantime that causes the undo to conflict, it won’t work. (Before transaction X, some value was A which X changed to B, but later something changes it to C. If I now want to undo transaction X to get back to A, it will conflict. Catalogs and other shared state are prime candidates).

But you still might be lucky and there won’t be a conflict. So, how do you undo a transaction? First, you need to find the transaction. In my case, I knew an object that had been changed by my script. So I asked the ZODB for the history of that object, i.e. the last transaction(s) that changed it:

>>> db = root._p_jar.db()
>>> hist = db.history(my_changed_object._p_oid)
>>> hist
[{'tid': '\x03...', 'size': 123, 'user_name': '', 'description': '', 'time': 1304493667.320477}]

Now I have the offending transaction’s ID. However, the undo() API does not work with transaction ids but needs a special (storage-specific) identifier. And, since as far as I can tell there is no way to map a transaction id to an “undo id”, I had to make to by matching the time stamp:

>>> info = db.undoInfo(specification=dict(time=hist['time']))
>>> info
[{'id': 'A44XmR876bs=', 'time': 1304493667.320477, 'user_name': '', 'description': '', 'size': 315893}]

Finally, call undo and hope you don’t get a conflict upon committing:

>>> db.undo(info['id'])
>>> import transaction
>>> transaction.commit()

Get a zc.sourcefactory to implement an interface

zc.sourcefactory is very handy to easily create a source (zope.schema.interfaces.IIterableSource to be precise) with corresponding titles and tokens for its contents. Every now and then a source requires an explicit interface. For zc.sourcefactory the following code snippet helps:

class IMySource(zope.schema.interfaces.IIterableSource):
    """my source"""

class MySource(
    zc.sourcefactory.contextual.BasicContextualSourceFactory):
    """The source factory."""

    class source_class(
        zc.sourcefactory.source.FactoredContextualSource):
        """This class is being instanciated by the factory.

        It *must* be called source_class.
        """
        zope.interface.implements(IMySource)

    def getValues(self, context):
        …

Of course it is also possible to declare the source_class separately from the source factory and reference it. But since its sole purpose is to hold an implements declaration, I’m fine with defining it inline.

Testing pyramid applications with zope.testbrowser

The Pyramid documentation recommends using WebTest for functional tests, but coming from the Zope world, zope.testbrowser is the more familiar tool for this job (and it seems a little more high-level than WebTest at first glance).

With the 3.11 release,  zope.testbrowser gained out-of-the-box support for talking to WSGI applications (courtesy of wsgi_intercept), so herewith some corresponding example code to show the basic usage and setup. (This assumes you are using zope.testrunner or something else that supports its layers).

import unittest
import zope.testbrowser.wsgi

class MyAppLayer(zope.testbrowser.wsgi.Layer):

    def make_wsgi_app(self):
        from myapp import main
        return main({})

MY_APP_LAYER = MyAppLayer()

class FunctionalTests(unittest.TestCase):

    layer = MY_APP_LAYER

    def test_root(self):
        browser = zope.testbrowser.wsgi.Browser()
        browser.open('http://localhost/')
        self.failUnless('Pyramid' in browser.contents)

Adding round corners to boxes using CSS and JavaScript

Curvycorners has a way to make gorgeous roundings to your boxes without having to split each box into pieces of background images. You simply need to include one JavaScript file and add two lines of CSS code to your box (or to the box’s class definition).

First, include the curvycorners javascript file in the head part of your webpage.

Then, add the following lines to the box that should get round corners. Here you can also customize the radius:

-moz-border-radius: 3px;-webkit-border-radius: 3px; 

That’s all, the result looks like this:

A box with curvy corners.

Localized dates with Zope 2 and Zope 3

Imagine, you want to display a date in a nicer way than just 2009-11-17. It’s not that difficult to localize it into e.g. German and output 17.11.2009. But what, if you need to display Mittwoch, 11.November 2009 instead?

Calling strftime(‘%A, %d.%B %Y’) on your date object would format the date with month- and week names, but in most cases display the names in English and not in German, as we @gocept need it.

Locales provide you with the nessecary functionality to format your datetime object in the users local format and language. You can choose from different templates (parameter length) or even create your own templates.

The example defines a method formatDate which takes a datetime object. It retrieves the users locale settings from the request, chooses a format template and then “mixes” everything together :-).

So, to get something like “Wednesday, 17.November 2009”, you will need to submit the length parameter full as shown in the above example:

class MyView(object):
def formatDate(self, datetime_date):
if not self.request._locale:
# zope2 doesn't initialize the locale itself
self.request.setupLocale()
formatter = self.request._locale.dates.getFormatter(
'date',  length='full')
return formatter.format(datetime_date)