18 Dexterity

A few years ago nearly all the code I wrote was in C++, but increasingly I’m finding myself writing in a variety of mostly C-style languages and having to perform crunching mental gear changes as I switch between them.

In the interests of making these language switches less painful I thought about listing the commonly used features of the languages I commonly use in a side-by-side format. Luckily I’m a lazy programmer, the web is large and there’s nothing new under the sun, so I quickly found Hyperpolyglot which provides commonly used programming language features in a side-by-side format, which is what I wanted. Nearly.

Hyperpolyglot organizes it’s language comparisons in to several catagories: scripting languages, C++ family languages, embeddable languages and so on. In my case (and I suspect in many cases) the languages I wanted to compare were spread across several pages.

After briefly considering some cut and paste to get what I wanted I started playing with Google Spreadsheets, which has a very nifty importHtml function which allowed me to pull the Hyperpolyglot data in to several sheets which can be combined to produce arbitrary language comparisons.

It’s not perfect as different languages have different features and in some cases the Hyperpolyglot data doesn’t use exactly the same terms across tables (“version used” vs “versions used”) and I’m not a spreadsheet ninja, but it’s good enough to generate PDFs like this JavaScript Python Java C++ Comparision. As a Hyperpolyglot derivative work, The Super Hyperpolyglot Spreadsheet is licensed under the Creative Commons Attribution-ShareAlike 3.0 License, please let me know if you improve it.

A couple of years ago I had great fun putting together the London Geek Community iPhone OSCestra at Open Hack London and I’ve been controlling Ableton Live with iPhone tapped to my guitar as part of 100 robots for a couple of years now so when @andybudd suggested I do a digital music thing for the Brighton Digital Festival I immediately thought that it would be fun to combine the 2 projects by doing a 100 robots performance with audience participation.

The iPhone OSCestra was effectively a distributed collaborative mixing desk with each person controlling the volume and effect parameters on one channel of a playing back Ableton Live set. For the 100 robots performance I wanted to go further and have the audience actually adding parts to the musical performance, so @toastkid and I added extra drum, bass, synth and sample tracks to the 100 robots live set and filled them full of samples that could be triggered by the audience.

While having the samples adjust in tempo to match each song was relatively simple, transposing them to match the key of each song was more complicated. First I built a custom slice to midi preset which mapped the sample transpose to a macro control and used it to slice all of the samples to MIDI tracks, then mapped all of the transpose controls to a single MIDI controller and added a MIDI track which output the appropriate controller value for each song to a MIDI output which was looped back in to Live to transpose the samples.

The next question was how to avoid the performance turning in to a mush if multiple drum tracks or bass parts were playing concurrently. To avoid this we put dummy clips on the normal 100 robots which muted the normal parts when the audience triggered parts were playing. In some cases we let the audience parts add to the music, in others the audience parts would play instead of the normal tracks.

A final question was how to avoid max and I getting lost when the normal parts we play along to were replaced by unfamiliar samples. To deal with this we set the clip quantization on the audience triggered clips to values longer than the clip length. This meant that even if alternate baselines were constantly being launched, we would still hear the normal bassline for a while at the end of each quantization period, so we would know where we were with the track. To tune these settings we did some fuzz testing with semi random MIDI data to see how much madness we could deal with and still manage to play the songs.

With the tests done it was time to perform with 100 robots and 100s of people at the Brighton Dome and Museum.

Many thanks to Steve Liddell for recording the Brighton Museum set, @aral for letting us experiment on his update conference and to everyone who participated and watched. If you’d like to host another performance, please get in touch and if you like the music, please check out the 100 robots blog and consider buying our album from bandcamp.

There was a slide in the early talks that Cory Ondrejka used to give about Second Life about alien abductions in Second Life. One of the most exciting moments in Second Life for the early Lindens was when a resident constructed a UFO and flew around the world abducting other residents and then returning them to the world with a commemorative t-shirt. It was exciting because it was unanticipated. The Lindens had created a virtual world that enabled interaction and someone had taken it and run with it to create a fun and engaging experience.

So, once I’d finished implementing a simple interest management and collision detection system for the Brighton Digital Festival JavaScript Jungle to enable interactions, I thought I would implement an alien abductor as a hat tip to Second Life.

The JavaScript first adds a UFO from You’re The Boss 2 to the supplied div along with an SVG canvas containing a hidden translucent tractor beam path before binding to the see and tick events. The tick handler implements a state machine which either moves the UFO towards a random spot, a target creature that the UFO has seen or drags the target off screen for diabolical experimentation.

The most interesting part of the code on line 155 which replaces the target’s position method with one which returns the target’s position, but doesn’t update. This allows the UFO to move the target while the position updates made by the target’s own code call the new read only position method. Tom Parslow‘s boids look especially mournful flapping around and turning towards the flock while being captured.

While the alien abductions in Second Life and the JavaScript jungle are meant to be fun and mostly harmless, the same mechanisms that enable them can be used for griefing in virtual environments and malware in software at large. The ability for scripted objects in Second Life to self replicate caused dozens of problems with grey goo attacks for every amazing virtual ecosystem and many malicious cage attacks for every playful alien abductor.

The message passing concurrency model adopted by LSL actually made direct attacks on other scripts of the kind used by the JavaScript Jungle UFO very hard, but things are much harder in JavaScript’s browser environment even when separating scripts in iFrames.

Luckily projects like Caja and Belay (which is being worked on by another ex-Linden, Mark Lentczner ) are working on the problem of making multiple scripts work safely in the same browser.

The challenge for sandboxes like Second Life and the JavaScript jungle is to allow interesting and meaningful interactions with emergent properties and unanticipated consequences without allowing malicious scripts to destroy that environment. Building the JavaScript Jungle was a lot of fun and made for another great Brighton Digital Festival project. Many congratulations to @premasagar, @ac94, @purge and everyone else for making it a success. Maybe next time we can try to build a secure JavaScript Jungle that is both secure and expressive.

One of the events that kicked off Brighton Digital Festival was dConstruct, the always thought provoking conference run by clearleft.

As usual I found most of the sessions interesting, but not always relevant as there’s a heavy design rather than development focus. The most relevant talk this year was Kevin Slavin’s final talk, Reality is Plenty, which argued that augmented reality is not the next big thing, just as it wasn’t in 2005.

Despite Kevin having a dig at Second Life and having spent a lot of time working on Augmented Reality with Blast Theory while at Nottingham University, I mostly agreed. While there are definitely use cases which benefit from augmented reality (fighter pilot navigation systems and things like Carbon Goggles which are all about making invisible aspects of objects visible) and virtual reality (simulation and virtual meeting spaces) there are plenty of others which are better served by other interfaces. Environments like Second Life are particularly exciting as they allow people to quickly prototype systems to discover which applications work and which don’t.

With both AR and VR it’s tempting to argue that they allow for intuitive interfaces as they model or overlay the real world: people know how to navigate a 3D space so they know how to use a 3D environment and they know how to use AR as they can see. Anyone who has done their time climbing the Second Life learning curve or trying to use AR to find their way around will know this clearly isn’t true. Apparently more abstract interfaces like maps, which talk to the mind rather than the senses are often much easier to use.

There’s a lot of work to be done to make both AR and VR as easy to use as 2D interfaces, let alone as natural as using real world senses. Now that the huge technical problems around networking virtual environments and tracking real world objects with mobile devices are starting to be solved, it is mostly UI work that needs to be done to make these technologies more widely used.

Even if the UX issues are solved there will still be many cases where speaking to the mind is much better than speaking to the senses.

Over the last couple of years I’ve found myself using python decorators to annotate handlers for web requests more and more, both when using Django and with micro-frameworks like mnml and newf.

Where the same functionality is required for all handlers, or the required functionality can be determined from standard request or response headers, using WSGI or Django middleware is fine, but where the required functionality is varies based on the handler its much cleaner to use a parameterised decorator than poluting the environment or response objects just to control the middleware. Functionality can be added to a framework as a suite of decorators and plugged together in an aspect oriented way like lego to easily build up sophisticated behaviours.

Unlike other mechanisms for implementing macros, templating or aspect orientation that introduce a new language, python decorators are pure syntactic sugar that under the hood are simply rewritten as python expressions:

@requires_oauth_scope("email")
def notify_friends(request):
    pass

Is simply shorthand for:

def notify_friends(request):
    pass

notify_friends = requires_oauth_scope("email")(notify_friends)

This simplicity is powerful as it allows decorators to also be used as normal functions, for example to build up higher level decorators that bundle common decorator configurations, but it also means that decorators potentially interact badly with another powerful Python feature: introspection.

In the above example the undecorated notify_friends function has the __name__ “notify_friends”, but the decorated function has the __name__ “requires_oauth_scope”. When decorators are used extensively, this can seriously impact the usefulness of introspection for debugging or generating documentation.

Decorating your decorators with the functools @wraps decorator, which copies the __name__ of the wrapped function over to the wrapping function solves this introspection problem, but introduces another: the decorators now become invisible to introspection. In the example above the __name__ of the decorated function would now be “notify_friends” as in the undecorated case, but we wouldn’t know that the function had been decorated or not.

A potential solution to this new problem is to store the details about the decoration in another attribute that can be inspected at runtime. In addition to copying over the __name__ attribute, functools.wraps also copies over the target __dict__ by default, allowing it to be used to store information about the decoration and be correctly copied over when decorators are chained:

from functools import wraps

def requires_oauth_scope(scope):

    def decorator(target):

        target.__dict__["my_project_requires_oauth_scope"] = scope

        @wraps(target)
        def wrapper(*args, **kwargs):
            # return target(*args, **kwargs) or FORBIDDEN if token does not have required scope

        return wrapper
    return decorator

By constructing decorators in this way we get the benefits of python decorators and more declarative C# style attributes that are visible to introspection.

After months of rumours it’s finally here, so what is Google+ like? My first thoughts are that it’s super slick and that Circles definitely makes it different, but I’m not sure better.

Limiting the distribution of shared information will likely also limit the growth of the network, something that’s not going to help Google+ grow to Facebook’s size. Will the limited sharing encourage more sharing? I’m not sure. There have certainly been a couple of times when I’ve held back from sharing inside Google Reader as I know not all of my Facebook friends will want to see it when it’s sent to my wall via FriendFeed but that doesn’t happen often.

When sharing inside Facebook directly I already have the choice of publishing to my friends via my wall, or to a particular group wall and I make that choice when I share. People have the choice of joining a group, or or adding me as a friend, or both. I choose where I publish when I publish and they choose what they subscribe to. Having to partition contacts up front, before I have anything to share, is harder. Maybe it will become more natural, but currently it feels weird. The sharing itself is also harder, feeling a lot more like composing an email, having to choose recipients than just a place to post.

Circles apart, Google+ definitely feels somewhat early and skeletal. Hangouts look like a nice group video chat feature, but it’s difficult to judge with so few people currently on the network. Sparks doesn’t look like it will ever be as useful as Google Reader to filter information on interests and, given the focus on social sharing, feels like a bootstrapping mechanism more than a long term feature. Mostly Google+ feels like a weird, empty, alternate reality version of Facebook with more primary colours, which is to be expected.

There’s a long way to go, but I’m glad Google has made a start. Google+ feels like it could be both a real competitor and alternative to Facebook, which is a good thing. One horse races are never very interesting to watch.

The slides and video of my talk at AsyncJS on Thursday are now online. The video is pretty murky, but the sound has come out fine and you can see enough of the slides to be able to follow along at home. The talk focuses on ways to bring useful software engineering patterns to JavaScript, patterns that will be increasingly important as JavaScript applications become larger and more complex.

Thanks to Prem for inviting me to talk and to everyone who came along to the Async session for the fascinating discussion.

Jamie has just uploaded the movie of my talk “The Why and How of Automated Testing with Python and Django” which I gave at BrightonPy a week ago (and this time it really is a movie, clocking in at a feature length 1 hr and 35 minutes). The audio on the video is fine (and arguably the laptop-eye-view video is improved by chopping my head off for large parts of the talk), but it’s tricky to see the slides on the video, so I’ve uploaded them to slideshare.

The talk rambles a bit in places and there are a couple of things that betray my static language roots for example you can’t actually use unit tests to discover dependencies as easily in python as you can in C++. I’m also already evolving the JS testing stack I talk about here: moving from qunit, qmock and Selenium to jsmockito and possibly JsTestDriver. Overall I think it’s a pretty good overview of how an agile software engineering process can be screwed together.

Many thanks to @garethr for donating his Fabric scripts, Spike for his database migration cameo, Si for recommending Hudson, Dave for hooking me on automated testing and j4mie for organising the night and wrangling the video. If you’d like me to help your organisation improve its agile engineering process, please get in touch.

Since leaving Linden Lab I have been talking to a number of people about doing freelance consulting and development work while I get my start-up off the ground and last week got round to setting up a UK limited company so that people will actually be able to pay me.

Setting up a company is insanely easy these days: if you go to companies made simple it will cost you less than £17 and 10 minutes of form filling. Coming up with a name is harder, but within a couple of hours I found that 18dex was available as a .com TLD, twitter account and facebook username. Meaningful 5 character .coms are pretty tricky to come by these days, so I snapped it up and 18 Dexterity Ltd. was born — a pretty fantastically geeky name for an agile software engineering company I hope you’ll agree.

A few minutes later I had a holding page up for 18dex.com, but it looked pretty sad with no content, so I started thinking about setting it up as a blog. I have a stack of relevant software engineering posts on jimpurbrick.com from the last few years, but they are sandwiched between less relevant posts on 100robots, Second Life and various miscellany. I didn’t want to move the software engineering posts from jimpurbrick.com as they’re part of what I do and regularly updating a single blog is quite enough work. I also didn’t want to copy the posts from one blog to another as it would potentially end up with 2 independent comment threads on each blog. There would be no definitive version of a post, a blatant violation of Don’t Repeat Yourself.

Luckily Django includes a piece of machinery to deal with this problem in its sites framework, something I’ve been meaning to have a closer look at for some time. The sites machinery simply lets you associate a piece of content with a site and keeps track of the current site, allowing you to filter the content in the database to only show a subset on each site.

While the byteflow blog engine I use for jimpurbrick.com supports the sites framework, each post is associated with a single site via a ForeignKey. In order to allow posts to be shown on both jimpurbrick.com and 18dex.com I had to change that ForeignKey field to be a ManyToManyField: a single line change in the python code, but something that requires a little wrangling to massage the existing data to fit the new model.

I’ve been using the excellent South in all my recent projects to allow me to easily migrate data across django model changes. Although jimpurbrick.com dates from long before South was available I managed to convince south to manage the migration by dumping the blog_post table to json, dropping the table and recreating it with south, reloading the data and then letting south migrate the data to the new ManyToMany schema. While this was slightly more fiddly than it could have been it means that the blog app is now being managed by south, which will make future development on the blogs much easier.

Once I had migrated the data to the new model and associated the software engineering posts in jimpurbrick.com with both sites in the django admin interface all that remained was for me to clone the jimpurbrick.com directory with mercurial to create an 18dex.com directory and choose and tweak a byteflow theme for the new site.

Once again I’ve been very impressed with Django and Byteflow, which have proven to be incredibly powerful tools that are very easy to work with. In a few hours I was able to create professional and personal views on to my blogging which can be easily administered from a single interface and allow comment threads and users to easily flow between them. If you’re just interested in my software engineering posts, head over to 18dex.com, if you want to hear about music, Second Life and everything else I get up to, stay subscribed to jimpurbrick.com. If you notice anything broken on either blog, then please leave a comment to let me know.

My recent experiments in to using Procssing.js and HTML5 audio to generate multimedia web applications didn’t get very far. I first tried generating a new HTML 5 audio element for each audio event, which quickly caused the browser to grind to a halt, and my attempts to reuse audio elements by resetting the playback position didn’t seem work, leading me to conclude that HTML 5 audio was only really useful for playing back long audio files, not for building sequencers that play back many short samples. When I spoke to @rem about my findings he was convinced that resetting audio elements should be possible and this weekend’s Music Hackday London has provided the perfect incentive and opportunity to dust off my experiments and start tinkering again. An hour in and sure enough I’ve managed to get audio elements to reset: it seems that the trick is to set currentTime after calling play() on the element, something that seems very counter-intuitive, but seems to work (at least in Firefox 3.6.8 and Safari 5.0.1 on OS X 10.6.4). Now I have reliable sample play back it’s time to start playing around with more interesting interfaces in Processing and there are 26 hours of hacking left: game on!

• Older posts