Yeoldeclue

Kamaelia components from decorated generators. Pythonic concurrency?

Michael — Sun, 04 Oct 2009 21:55:16 +0000

A few months ago, there was a thread on the then google group python-concurrency about some standard forms for showing how some libraries deal with concurrent problems. The specific example chosen looked like this:

#!/bin/sh
tail -f /var/log/system.log |grep pants

Pete Fein also posted an example of this using generators, based on David Beazley's talk on python generators being used as (limited) coroutines:

import time
import re

Restarting Python Northwest. 24th Sept

Michael — Mon, 14 Sep 2009 22:36:42 +0000

Traffic Server to be Open Source?!

Michael — Tue, 07 Jul 2009 11:02:45 +0000

If this happens this will be awesome. Traffic Server is some really nice code. It's a large codebase, but it's really cool, and it *scales*. (I used to work at Inktomi, so have been inside the code as well). For those that don't know what it is, it's a very high performance web caching proxy, with a plugin architecture, allowing for the addition of other protocols. It used to support HTTP (& obvious friends), NNTP, RTSP, RTP, WMV, etc.

Europython Videos Transcoding

Michael — Tue, 07 Jul 2009 00:04:31 +0000

Since I've had a few questions about this, a short status update. At Europython last week I was recording all the talks I was attending. Including the lightning talks this means I have video from 55 talks. The video files from the camera are too large for blip.tv, so I'm transcoding them down to a smaller size, before uploading them. Since these 55 talks are spread over nearly 80 files, that naturally takes time.

Autoloading in python

Michael — Sun, 21 Jun 2009 15:15:00 +0000

Before I started using python, I'd used perl for several years, and one thing which I'd liked about perl was its autoload facility. Now in python the closest equivalent that I've seen is __getattr__ for classes, but not __getattr__ for a module. This seemed like a real shame since there are times when autoload can be incredibly useful.
If it seems chaotic, consider the Unix PATH variable. Any time you type a name, the shell looks in lots of locations and runs the first one it finds. That's effectively the same sort of idea as autoloading. Yes, you can do some really nasty magic if you want, but then you can do that with the shell to, and generally people get along find.
Anyway, vaguely curious about it I decided to do some digging around, and came across this post by Leif K Brookes, which suggests this:

You could wrap it in an object, but that's a bit of a hack.

Europython

Michael — Wed, 10 Jun 2009 16:35:47 +0000

I've mentioned this in a couple of places, but mentioning on my blog seems appropriate too.

New Tool for helping debugging Kamaelia Systems

Michael — Mon, 08 Jun 2009 17:17:59 +0000

I've added a PythonInterpreter component to Kamaelia in Kamaelia.Experimental.PythonInterpreter. The primary use for this is intended to assist with debugging, but there are other possible uses.

Use in a system

This allows a variety of things, from a basic command line console:

StandaloneInterpreter().run()

Which you can run in the background on any console. For example you could do
this:

ServerCore(protocol=myprotocol, port=1234).activate()
StandaloneInterpreter().run()

Alternatively, you can use an embeddable component that speaks to inbox/outbox rather than stdin/stdout. Crudely you can do something like this:

Pipeline(
ConsoleReader(),
InterpreterTransformer(),
ConsoleEchoer(),
).run()

But you can also put it inside a pygame application, reading & writing from a
Textbox/TextDisplayer:
Pipeline(
Textbox(size = (800, 300), position = (100,380)),
InterpreterTransformer(),
TextDisplayer(size = (800, 300), position = (100,40)),
).run()

Going from generator coroutines to Kamaelia Components

Michael — Sun, 17 May 2009 01:10:56 +0000

Earlier this evening an announcement by Pete Fein regarding the formation of a python-concurrency mailing list (aka Python Concurrency Special Interest Group) bounced past my inbox in the unladen-swallow mailing list. Naturally, given my work on Axon (a concurrency library) and Kamaelia (a bunch of components that use it), it jumped out at me as interesting. (5 minute overview for those that don't know what Kamaelia is...)

Bar Camp Leeds UK

Michael — Tue, 05 May 2009 20:56:42 +0000

Bar Camp Leeds (UK) is running for its 3rd year running. It'd be great to see other Python people there if you're around. Details:

May 30th, May 31st
Old Broadcasting House, Leeds, UK
Website: http://barcampleeds.com/

Schedulers matter

Michael — Wed, 04 Mar 2009 10:26:54 +0000

Improving the scheduler. It's been something we've avoided for quite a while in Kamaelia, but Simon Wittber's recent post benchmarking Fibra vs Kamaelia vs stackless is interesting. His key showing that "Stackless is 7x faster than Fibra, and Fibra is 10x faster than Kamaelia" are cool for him :-) ... and naturally led me to think "why". Looking at the code, it struck me that he's doing something more interesting with the scheduler given these code forms:

scheduler.install(self.messageLoop())
# self.MessageLoop is a regular python generator
...
yield self.incrementCounter()
yield kickTo.messageQueue.push(self)

If you delve inside the fibra scheduler (which doesn't appear to be here unexpectedly) you see the following core:

    def tick(self):
        """Iterates the scheduler, running all tasks and calling all
        handlers.
        """
        active = False
        for handler in self.handlers:
            active = handler.is_waiting() or active
        active = (len(self.tasks) > 0) or active
        tasks = []
        while True:
            try:
                task, send_value = self.tasks.pop(0)
            except IndexError, e:
                break
            try:
                if isinstance(send_value, Exception):
                    r = task.throw(send_value)
                else:
                    r = task.send(send_value)
            except StopIteration, e:
                r = e
            if r is None:
                tasks.append((task, None))
            else:
                try:
                    handler = self.type_handlers[type(r)]
                except KeyError:
                    raise RuntimeError("Don't know what to do with yielded type: %s" % type(r))
                handler.handle(r, task)
        self.tasks[:] = tasks
        return active

The core of this appears to be "when I'm done, do this later" next. If you think that's familiar, it should be - its very similar (at least conceptually) to what twisted does with deferreds. It's not identical, but similar. So what does this mean for the hacksack demo? Well, if we look at self.tasks after each run, by changing:

    def run(self):
        while self.tick():
            pass

to:

    def run(self):
        while self.tick():
            print "TASKS", [x[0] for x in self.tasks]

It becomes apparent what's happening (though it's fairly obvious from above):

TASKS []
TASKS []
TASKS []
TASKS []
TASKS []
TASKS []
TASKS []
TASKS []
TASKS []
TASKS []
TASKS []
TASKS []

Fundamentally, the reason it's quicker for two reasons:

It always knows which generator is ready to run next.
It also defaults to pausing the generator, unless it specifically asks to be run. ie the tasks default to being passive.
The sender knows who the receiver is. This allows the sender to schedule the receiver explicitly.

These two points matter because Axon's scheduler (from jesse's post, hopefully people are aware that Axon is the relevant part of kamaelia here):

Is effectively a round robin scheduler - essentially for simplicity. The key question we wanted to ask was "does a component model based around inboxes/outboxes make it easier to make easier to write/reuse concurrent software", rather than "how can we build a faster, more responsive scheduler". As a result the round robin scheduler was always a compromise. There's a little bit of intelligence in there, but not much.
Kamaelia's components default to active, rather than passive. That is they're expected to run continuously unless explicitly paused. This design decision impacts the scheduler as a whole.
The sender does not know who the receiver is. This requires something different to happen in the scheduler. On the upside, it means that code can be tested easier in isolation, or reused in situations it wasn't originally expected to be used within.

This leads me to the rather obvious solution here - can we rebuild a better Axon scheduler by reusing fibra and throw away our scheduler? If so that would be really neat - throwing away our scheduler for something faster and more intelligent would be rather fun :) If we can't then we've been pondering improving it - that is making it more intelligent - for a while. Fibra's benchmarks suggest that doing so would be well worth it. The question this raises though is whether doing this would help us in the general case. At present I'm unclear on that, but until you try, you don't know.