Manage JavaScript dependencies with BowerStatic

Last month I explained how to use Fanstatic to manage JS dependencies. Since we were more and more displeased by using Fanstatic, we recently switched to BowerStatic, the new kid on the block. Since the setup is a bit more complicated and you need more tools to have the same features as you had with Fanstatic, I will describe how to set up the full toolchain. After that, I will shortly summarize the benefits and drawbacks of BowerStatic, so you can decide for yourself if you prefer BowerStatic above Fanstatic or the other way around.

Of course the choices made in the toolchain of BowerStatic are strongly linked to other tools we use. Since we set up large projects using Buildout, we like to use Buildout recipes to solve a problem. Therefore this blogpost is most interesting when you are using Buildout yourself.

How it works

BowerStatic was created by Martijn Faassen, who was a core developer of Fanstatic. We think of BowerStatic as a reimagined version of Fanstatic, avoiding disadvantageos decisions made in the past. One problem of Fanstatic was, that it tried to do too many things: Fetching the files, delivering them to the client and bundling them into one large file.

Therefore BowerStatic has a narrow focus: It only helps to manage which libraries should be delivered to the client. It does not care how the files were retrieved or whether you want to have a bundled resource. To automate these steps as well, you must look for additional tools.

We use the following toolchain:

  • a Buildout recipe for Bower to download all external libraries
  • BowerStatic to deliver JS files to the client
  • a custom Buildout recipe to bundle all files of the project into one large file

Since you need all of these tools to have the same features as Fanstatic, I will give a short description, how to set things up.

Bower

Bower is a small tool to download libraries and their dependencies by using a simple description of the library in JSON, which is named bower.json. The description contains a version string, the path to contained files, a list of libraries it depends on and a lot more metadata like author, homepage, keywords and so on. For example:

{
  "name": "bootstrap",
  "version": "3.3.2",
  "main": [
    "less/bootstrap.less",
    "dist/css/bootstrap.css",
    "dist/js/bootstrap.js",
    "dist/fonts/glyphicons-halflings-regular.eot",
    "dist/fonts/glyphicons-halflings-regular.svg",
    "dist/fonts/glyphicons-halflings-regular.ttf",
    "dist/fonts/glyphicons-halflings-regular.woff"
  ],
  "dependencies": {
    "jquery": ">= 1.9.1"
  }
}

The great thing about Bower is, that you just point to a github repository in case this certain library is not registered on the Bower server. In this case it will check out the trunk of the repository and creates the bower.json all by itself using the commit hash as it’s version. Of course it cannot auto-detect dependencies, which are normally declared in the bower.json.

It’s even better if the repository contains a bower.json file. In this case Bower will only load the files mentioned there and can retrieve dependencies. Therefore it makes no difference if you point Bower to a registered package, e.g. jquery-ui, or if you point it to the github repository of this project, e.g. github.com/jquery/jquery-ui. Of course the version will be different (stable release vs trunk).

To download external dependencies via Bower, we added bowerrecipe to our Buildout configuration. This way we can just list the libraries and their version in the recipe configuration. Bower will automatically download those libraries, as well as their dependencies. The following example will download Bootstrap and jQuery, since Bootstrap depends on jQuery.

[bower]
recipe = bowerrecipe
packages =
    bootstrap#3.3.x
executable = ${buildout:bin-directory}/bower

By default the libraries will be stored inside parts/bower/downloads. The destination can be changed inside the recipe configuration.

BowerStatic

BowerStatic uses libraries downloaded via Bower to resolve JS dependencies and delivers all required files to the client. To do so, it must be registered as a WSGI middleware. Compared to Fanstatic, the setup of BowerStatic is a little bit more complicated:

  1. Create an instance of the Bower class
  2. Create a external component collection, which contains all external libraries loaded into the directory parts/bower
  3. Create a local component collection, which will contain your self-written JS files
  4. Set the external collection as the fallback of your local collection, i.e. JS libraries not found in the local collection will be looked up in the external collection
  5. Add your self-written files to the local collection, by reading a bower.json at given path

This may sound like voodoo to you. The documentation of BowerStatic contains some insightful statements why the setup is so complicated. Written in code the setup will look like this:

import bowerstatic
import os
import pkg_resources

bower = bowerstatic.Bower()
external_components = bower.components(
    name='components', path='parts/bower/downloads')
local_components = bower.local_components(
    name='local', component_collection=external_components)

local_components.component(path=pkg_resources.resource_filename(
    'my.cool.package', 'resources'), version=None)

Let’s assume in my.cool.package.resources is a bower.json like:

{
  "name": "custom",
  "version": "0.1",
  "main": [
    "custom.js",
    "custom.css",
  ],
  "dependencies": {
    "bootstrap": "*"
  }
}

Then you have the JS library called custom in your local component collection. After handling this setup, you can use BowerStatic similar to Fanstatic, by including JS files inside any view. However, you always need the component collection to do so. local_components.include(self.request.environ, 'custom') will load custom.js, custom.css, the bootstrap library and the jQuery library. You could also write local_components.include(self.request.environ, 'jquery'), which would only load jQuery.

Bundling

Since we were used to Fanstatics automatism to bundle and minify all resources, we wanted to have something similar with BowerStatic. It is not sure if BowerStatic will ever contain bundling mechanisms, since this may be out of the narrow scope it wants to preserve.

Therefore we build our own Buildout recipe, gocept.recipe.bowerstaticbundler. This recipe will create another Bower library in parts/bower, which contains a minified JS file, a minified CSS file and assets like images and fonts. By using the dependency mechanism of BowerStatic, all JS files are bundled in the right order.

To use the bundled library in production and the separate files in development mode, we build a custom include with an environment variable switch:

def bower_include(environ, name):
    if not os.environ.get('BOWERSTATIC_DEBUG'):
        name = 'bowerstatic_bundle_' + name.replace('.', '_')
    include = local_components.includer(environ)
    include(name)

Of course you must use this include method instead of using the BowerStatic component directly, i.e. replace alo occurences of local_components.include(environ, name) with bower_include(environ, name).

This solution is good enough for ourselves, but may not be ready for your purposes. We have build it with a certain project in mind, so you might come across bugs when using it with different JS libraries. Since we will soon use it in production, we will happily fix bugs you report and merge tested improvements. You can find the code and issue tracker on bitbucket.

Benefits

The combination of Bower, BowerStatic and a bundling mechanism has the same benefits as Fanstatic, i.e. that you no longer need to copy JS files into your project and link them statically, as well as the benefits of bundled resources to speed up the initial page load.

The biggest contrast of these solutions is how external libraries have to be prepared: With Fanstatic you had to build an integration package for each library and release a version on PyPI for each version of the library. With BowerStatic it’s enough to have a current bower.json inside the library. So both need some kind of work and maintenance.

However, with Fanstatic the assumption was that someone else already built the integration package for you, so that you had to do nothing. As described in the Fanstatic article from last month, this assumption is no longer valid and many JS libraries are not supported or out of date. Updating a foreign integration package consumes time and may not be possible at all, in case the user is no longer active.

With BowerStatic, you can simply point to a git repository and select the version by using a certain commit or tag. As mentioned before, Bower will download the trunk of the repository and generates a simple bower.json automatically. Therefore every JS library that is hosted on github can be used with BowerStatic without additional work. This means no integration packages anymore, yay!

The downside is that you must declare dependencies of those libraries as your dependencies, since Bower cannot detect dependencies when it automatically creates a bower.json from the trunk of the repository. However, if there is a bower.json inside the repository, Bower will use it and to resolve dependencies.

The possibility of Bower to download a github repository comes with the benefit that it is really easy to try out different libraries for the same task. With Fanstatic you had to build an integration package for every experiment or copy the files into your resources. Now you can just point to the github repository and give it a try. Fanstatic always felt like a big hindrance when trying out new libraries.

Drawbacks

In my opinion the biggest drawback of BowerStatic is the long toolchain. You have to understand how Bower works, how BowerStatic uses Bower and how to set it up. Building a custom recipe for bundling was a bit time consuming as well.

After about 2 weeks we had aggregated enough wisdom to use the toolchain smoothly. We think of it as an investment in the future, since we were so annoyed by Fanstatic, building one integration package after the other.

Another big downside is, that there is no obvious solution to bundling when using BowerStatic. Of course you could bundle the resources using Grunt or any other approach of your choice. But we wanted a solution that fits into our toolchain, i.e. a solution that works with Python and integrates into Buildout. We did not know of any solutions that matched this criteria, therefore we created the Buildout recipe ourselves. However, building the recipe consumed some time and it is nowhere near “complete”. I hope that there will be a bundling solution that integrates nicely with BowerStatic in the near future. Till then, the Buildout recipe is good enough.

Conclusion

Wow, that was a long article. I hope it was interesting for you to read about BowerStatic. Chances are that you did not know about it before. Setting it up did take some time, but maybe the code snippets above help you to speed things up.

All in all I am really glad that we switched to BowerStatic. After the initial learning investment, you know almost everything that you need to know for the future. Testing brand new JavaScript libraries is a breeze. Even using an unknown library takes no additional effort. Sounds like a win to me.

Manage JavaScript dependencies with Fanstatic

Until the beginning of this year, we were using Fanstatic to manage dependencies to external JavaScript libraries. In case you are not familiar with Fanstatic, here is a short overview. I will discuss benefits and drawbacks later on.

How it works

Imagine you want to use jQuery in one of your projects. Therefore you build an integration package, say js.jquery, which contains the jquery.js file. You also add a resource.py which would look like this:

from fanstatic import Library, Resource

library = Library('jquery', 'resources')
jquery = Resource(library, 'jquery.js', minified='jquery.min.js')

To finish up, you state the version of the downloaded jquery.js in the setup.py and upload the package to PyPI.

After wrapping your app with Fanstatic, you can declare JavaScript dependencies by listing the integration package inside your setup.py. To use them, you write js.jquery.need() inside any view. Of course you can also add a resource.py and declare a local resource with dependencies:

import fanstatic
import js.jquery

library = fanstatic.Library('custom', 'resources')
my_awesome_js_code = fanstatic.Resource(
    library, 'js/my_awesome_js_code.js',
    minified='js/my_awesome_js_code.min.js',
    depends=[js.jquery.jquery])

Every time you call my_awesome_js_code.need() inside a view, it will also load jQuery.

Benefits

With Fanstatic you no longer need to copy the JavaScript files inside your project and link them statically, you just declare them as a dependency to your own JavaScript code and call my_awesome_js_code.need() inside a view.

The preparation described above can be tedious, but the assumption is, that someone else has built the integration package before, so you can reuse it. Installing a dependency therefore is as easy as adding a line to your setup.py and declare it as a dependency of one of your resources.

Fanstatic makes it a lot easier to update external libraries, since you only need to increase the version number in your setup.py. You can also make sure to use the same version across many projects, e.g. by sharing some buildout configuration across projects.

And besides many small nice features, Fanstatic can also create bundles, i.e. it will merge my_awesome_js_code and all of its dependencies into one big file and deliver it to the client, rather delivering all files separately. This usually makes the initial page load much faster.

Drawbacks

Despite all the benefits Fanstatic offers, we were getting more and more frustrated with it. The main reason is, that we feel a decline in the activity of the Fanstatic user group. With fewer people using Fanstatic, we often have to build the integration packages ourselves. This eliminates one of the main benefits, i.e. that we could reuse the integration packages of others.

For example, in a big project we did last year, we used integration packages for js.jquery, js.classy, js.chosen and many more. Those are pretty popular packages and the integration worked out of the box.

However, even those packages have issues: The integration package for jQuery does not offer version 2.0 or higher and Chosen is only available in a single (outdated) version. We often stumbled upon similar issues with other libraries, e.g. the integration package for js.modernizr is more than 2 years behind.

In addition, some newly developed libraries may not be available at all. Of course we could write the integration packages for those or update the jQuery integration package. But this would mean to find out where the code for an integration package is hosted, write a pull request and wait for a merge, as well as a new release on PyPI. In case the owner is not active anymore, you are out of luck and must add a duplicate package to PyPI. Tedious and unsatisfactory.

This is why we started to look for an alternative to Fanstatic. And maybe we have found it, but this is a story for another day, where I will talk about our experiences with Bower and Bowerstatic.

Notizen aus dem Norden

Unsere Herberge auf Rügen 2012
Unsere Herberge auf Rügen 2012

Heute neigt sich unser diesjähriger Sprint dem Ende. Am Dienstag sind wir gen Norden auf die Insel Rügen gefahren, um uns für 3 Tage intensiv darüber zu unterhalten, wie wir in Zukunft Software entwickeln wollen. Der Fokus lag dabei auf der Client-Seite, also dem Teil der Anwendung, der im Browser ausgeführt wird. Hier wollen wir weitere Erfahrung sammeln und Konzepte suchen, wie man wart- und testbaren, strukturierten Code schreibt.

Im Vorfeld haben wir uns auf drei JavaScript Frameworks/Bibliotheken geeinigt, die wir uns dann am Mittwoch in drei Gruppen à zwei Personen intensiv angeschaut haben: Ember, Obviel und Backbone. Neben Dokumentation lesen und Tutorials ausprobieren haben wir auch gegen einen selbst-gestrickten “Spielserver” programmiert, um durch die praktische Erfahrung ein Gefühl für die Konzepte hinter dem Framework zu bekommen und einschätzen zu können, inwieweit sich das Framework in unserem Alltag einsetzen lässt.

Ember hat uns für den Einsatz bei “Grüne Wiese”-Projekten begeistert. Durch die strikte Trennung von Model, View und Controller (MVC) schreibt man nicht nur automatisch saubereren, sondern auch testbaren Code. Das Objekt-Modell bringt automatisierte, bidirektionale Aktualisierung zwischen Model und View mit, und für die Speicherung auf dem Server gibt es ein kleines, gut integriertes REST-Plugin. Der Aufwand, eine funktionierende CRUD-Anwendung, die Änderungen automatisch an den Server propagiert, zu entwickeln, ist mit Ember erstaunlich gering. Auch wie Ember es erlaubt, seine Klassen zu beerben, ist uns positiv aufgefallen. Leider haben wir es nicht geschafft, zwei Ember-Anwendungen auf einer Seite zu laden. Auch ist die Dokumentation noch recht dünn, was daran liegen dürfte, dass Ember noch ein junges Produkt ist.

Hilfreiche Links zu Ember:

Obviel ist streng genommen unser eigenes “Kind”, denn erste Ideen dazu entstanden im Jahre 2008 bei gocept. Obviel ist ein Client-seitiges Web-Framework für jQuery mit einem einfachen, aber genialen Prinzip: für ein JavaScript-Objekt rendert man einen View in einem Element. Dreh- und Angelpunkt bei Obviel sind die Interfaces, die definieren, wie ein View Objekte, die das Interface implementieren, rendert. Entwickler, die mit der Komponenten-Architektur von Zope vertraut sind, werden keine Probleme haben, Obviel zu verstehen. Weiterhin lässt sich Obviel problemlos in bestehende Projekte integrieren, es enthält Unterstützung für Internationalisierung (i18n), Forms mit Validierung, verschiedene Template-Engines, Subviews, …

Hilfreiche Links:

Backbone unterstützt den Entwickler bei der Strukturierung seiner Webanwendungen durch in Code gegossene Konventionen. Es wirkt sehr fokussiert, denn es tut nicht wirklich viel, aber das Wenige tut es sehr sinnvoll. Backbone ist, im Gegensatz zu Ember, sehr gut dokumentiert, löst aber im Moment keines unserer Probleme. Da Backbone insbesondere keine Unterstützung für data binding bietet, haben wir noch Knockout und Rivets ins Spiel gebracht. Für ersteres steht mit Knockback der „Glue-Code“, um die Frameworks miteinander zu verheiraten, schon zur Verfügung, für letzteres mussten wir ihn uns aus der Dokumentation und dem Issue-Tracker des Projekts zusammensuchen.

Weiterführende Links:

Ein Fazit zu ziehen fällt uns nicht leicht: Ember behalten wir für das nächste “Grüne Wiese”-Projekt auf jeden Fall im Auge. Da uns Backbone und Knockout/Rivets nicht überzeugen konnten, haben wir uns noch spontan AngularJS angeschaut – aber auch hier ist im Moment die Ernüchterung groß: Persistieren von Daten ist ein ungelöstes Problem, die Watcher auf dem Model (data binding) feuern nur unregelmäßig oder viel zu oft. Nach längerem Probieren haben wir herausgefunden, dass wir eine eigene Direktive schreiben müssen, die sich um die Persistierung kümmert.

Bleibt also Obviel, das wir an der einen oder anderen Stelle in unseren Projekten schon verwenden. Es wird wohl auch in Zukunft unser Werkzeug, um bestehenden JavaScript-Code zu verbessern oder neue Frontend-Einheiten zu bauen.

Das gocept-Team

Adding round corners to boxes using CSS and JavaScript

Curvycorners has a way to make gorgeous roundings to your boxes without having to split each box into pieces of background images. You simply need to include one JavaScript file and add two lines of CSS code to your box (or to the box’s class definition).

First, include the curvycorners javascript file in the head part of your webpage.

Then, add the following lines to the box that should get round corners. Here you can also customize the radius:

-moz-border-radius: 3px;-webkit-border-radius: 3px; 

That’s all, the result looks like this:

A box with curvy corners.