Viewing scales metrics from Pyramid

We’ve recently started experimenting with the excellent scales library to collect in-process metrics (see Coda Hale’s CodeConf talk “Metrics everywhere” among many others for reasons why one definitely wants to do that).

Scales comes with a flask-based HTTP server that allows viewing the collected measurements and dumping them as JSON. But if you already are in a web application, there’s no real need to spin up yet another thread, open another port etc. to do this. In our case, we’re using Pyramid, so here’s a quick recipe to get the same view that greplin.scales.flaskhandler provides:

Update 2013-11-06: This code is now released as pyramid_scales.

# in your Pyramid setup
config.add_route('scales', '/scales/*prefix')

from StringIO import StringIO
from pyramid.view import view_config
import greplin.scales
import greplin.scales.formats

@view_config(route_name='scales', renderer='string')
def scales_stats(request):
    parts = request.matchdict.get('prefix')
    path = '/'.join(parts)
    stats = greplin.scales.util.lookup(greplin.scales.getStats(), parts)

    output = StringIO()
    outputFormat = request.params.get('format', 'html')
    query = request.params.get('query', None)
    if outputFormat == 'json':
        request.response.content_type = 'application/json'
        greplin.scales.formats.jsonFormat(output, stats, query)
    elif outputFormat == 'prettyjson':
        request.response.content_type = 'application/json'
        greplin.scales.formats.jsonFormat(output, stats, query, pretty=True)
    else:
        request.response.content_type = 'text/html'
        # XXX Dear pyramid.renderers.string_renderer_factory,
        # you can't be serious
        request.response.default_content_type = 'not-text/html'
        output.write('<html>')
        greplin.scales.formats.htmlHeader(output, '/' + path, __name__, query)
        greplin.scales.formats.htmlFormat(output, tuple(parts), stats, query)
        output.write('</html>')

    return output.getvalue()

developer & admin BBQ IV

Our fourth BBQ (invitation post) had the most participants so far, almost 20 people were here to talk shop, exchange ideas and brave the unfortunately slightly rainy weather (the grilled goods were delicious regardless). We’re especially glad that the ratio of gocept people to guests was only about 50% this time, and we’re hoping it will go down further. 🙂

The sessions in the Open Space were about diverse subjects, ranging from “Deploying lots of Rasperry Pi’s” over “Gamification in a business context” to “Why is there no slim and simple CMS yet?”. In several sessions we didn’t find a satisfactory solution to the problem, but sometimes sharing your frustrations with others who have similar experiences is helpful in itself.

Since the session about code katas was very well liked, we’re thinking about maybe doing a Code Retreat instead of a classic Open Space for the next BBQ, so stay tuned.

Upcoming talks at PyCon DE 2012

Most of gocept is headed towards the German PyCon next week! We’re excited and look forward to meeting people, exchanging ideas and learning new things.

We have prepared several talks to share some of the experiences we gained over the last year:

See you in Leipzig!

Autumn Developer BBQ

Our second meet/talk/barbecue event was last week, and we think it was even better than the first one.

In the unconference part, these were the topics we talked about:

  • Demo of batou, the deployment tool
  • Django
  • Sentry: centralized error-logging
  • DRBD vs. NFS vs. Sheepdog
  • Beautiful Javascript

Thanks to everyone who joined us, we’re looking forward to meeting again in December! (We’ll probably have punch instead of barbecue then…)

Our first developer BBQ

We invited developers and sysadmins to join us for talking shop and barbecuing last Friday. Even though several people had signed up and said they wanted to come, at first none of the guests seemed to arrive. But after an hour and braving some ugly traffic jams on the way here, a few did make it. We’re happy you came to visit us, guys!

In the unconference part, these were the topics we talked about:

We had a good time (and the BBQ was tasty), so we’ll definitely want to do something like this again; here’s to hoping some more people will join us next time!

Profiling class-based views

Just a quick note for profiling e.g. Zope views:

class MyView(object):
    def __call__(self):
        result = {}
        cProfile.runctx('result["x"] = super(Body, self).__call__()',
                        globals(), locals(), '/tmp/viewprofile')
        return result['x']

Even though “exec ‘result = super(…) in globals(), locals()’ works, it seems that cProfile does something a little differently here, so that writing to a local variable is not possible.

Sprint fruits: gocept.exttest and gocept.package

The whole company spent three days in Kloster Drübeck sprinting on internal tools and topics.

We overhauled our workflow for generating invoices and identified steps that we could automate.

We polished and released gocept.exttest, which integrates for example JavaScript unittest to Python’s unittest framework. In a nutshell, it allows you to write JS tests like this:

require 'my_app.js'

describe 'MyApp', ->
  it 'has read Douglas Adams', ->
  expect(new MyApp().calculate_the_answer()).toEqual(42)

and include them into your Python test suite with a single command:

import gocept.exttest
def test_suite():
    return gocept.exttest.makeSuite(
        os.environ.get('jasmine-bin'),
        '--coffee',
        '--json',
        pkg_resources.resource_filename('your.package', 'tests'))

The third area of our efforts was documentation, we designed a Sphinx skeleton to make it easy to get started writing docs, and created a template for eggs that contains the necessary boilerplate and codifies our packaging and documentation conventions. While the concrete details are probably a bit specific to our tastes, some of the general mechanincs might be interesting to others, so we’ll release gocept.package once we’ve got the missing integration tests sorted out.

Getting sys.path out of buildout

A wrinkly part of buildout‘s design is that the PYTHONPATH is not easily available outside of scripts generated by buildout itself.

I’ve been using the following workaround in some of my development tools for a while and found it quite helpful, even though it’s hacky and rough around the edges:

#!/bin/bash
if [ ! -e bin/wpy ]; then
    bin/buildout install wpy
fi
export PYTHONPATH=$(bin/wpy -c "import sys; print ':'.join([p for p in sys.path if not p.startswith('/usr')])")
# do real work here

This needs a part that installs a python interpreter into the buildout; I’m using this in my ~/.buildout/default.cfg, since all projects I deal with have a [test] part:

[wpy]
recipe = zc.recipe.egg
eggs = ${test:eggs}
interpreter = wpy

A variation on this theme is this script which generates a TAGS file without the need for a special recipe (like z3c.recipe.tag) just to get the PYTHONPATH:

#!/bin/bash
if [ ! -e bin/wpy ]; then
    bin/buildout install wpy
fi
export IFS=
PATHS=$(bin/wpy -c "import sys; print '\n'.join([p for p in sys.path if not p.startswith('/usr')])")
echo $PATHS | ctags --python-kinds=-i -R -e -L -

Assertion helper for zope.testbrowser and unittest

zope.testbrowser is a valuable tool for integration tests. Historically,  the Zope community used to write quite a lot of doctests, but we at gocept have found them to be rather clumsy and too often yielding neither good tests nor good documentation. That’s why we don’t use doctest much anymore, and prefer plain unittest.TestCases instead. However, doctest has one very nice feature, ellipsis matching, that is really helpful for checking HTML output, since you can only make assertions about the parts that interest you. For example, given this kind of page:

>>> print browser.contents
<html>
  <head>
    <title>Simple Page</title>
  </head>
  <body>
    <h1>Simple Page</h1>
  </body>
</html>

If all you’re interested in is that the <h1> is rendered properly, you can simply say:

>>> print browser.contents
<...<h1>Simple Page</h1>...

We’ve now ported this functionality to unittest, as assertEllipsis, in gocept.testing. Some examples:

self.assertEllipsis('...bar...', 'foo bar qux')
# -> nothing happens

self.assertEllipsis('foo', 'bar')
# -> AssertionError: Differences (ndiff with -expected +actual):
     - foo
     + bar

self.assertNotEllipsis('foo', 'foo')
# -> AssertionError: "Value unexpectedly matches expression 'foo'."

To use it, inherit from gocept.testing.assertion.Ellipsis in addition to unittest.TestCase.

Shutting down an HTTPServer

For integration tests it can be helpful to have a fake HTTP server whose behaviour the tests can control. All necessary building blocks are even included in Python standard library. However, the BaseHTTPServer is surprisingly hard to shut down properly, so that it gives up the socket and everything.

While working on gocept.selenium, we came up with some code that does the trick (together with Jan-Wijbrand Kolman and Jan-Jaap Driessen).

class HTTPServer(BaseHTTPServer.HTTPServer):

    _continue = True

    def serve_until_shutdown(self):
        while self._continue:
            self.handle_request()

    def shutdown(self):
        self._continue = False
        # We fire a last request at the server in order to take it out of the
        # while loop in `self.serve_until_shutdown`.
        try:
            urllib2.urlopen(
                'http://%s:%s/' % (self.server_name, self.server_port))
        except urllib2.URLError:
            # If the server is already shut down, we receive a socket error,
            # which we ignore.
            pass
        self.server_close()

You might use this in a zope.testrunner layer like this:

class SilentRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):

    def log_message(self, format, *args):
        pass


class HTTPServerLayer(object):

    host = 'localhost'

    def setUp(self):
        self.server = None
        self.port = random.randint(30000, 40000)
        self.start_server()

    def start_server(self):
        self.server = HTTPServer((self.host, self.port), SilentRequestHandler)
        self.server_thread = threading.Thread(
            target=self.server.serve_until_shutdown)
        self.server_thread.daemon = True
        self.server_thread.start()
        # Kludge: Wait a little as it sometimes takes a while to get the server
        # started.
        time.sleep(0.25)

    def stop_server(self):
        if self.server is None:
            return
        self.server.shutdown()
        self.server_thread.join()

    def tearDown(self):
        self.stop_server()