Yeoldeclue

Autoloading in python

Michael — Sun, 21 Jun 2009 15:15:00 +0000

Before I started using python, I'd used perl for several years, and one thing which I'd liked about perl was its autoload facility. Now in python the closest equivalent that I've seen is __getattr__ for classes, but not __getattr__ for a module. This seemed like a real shame since there are times when autoload can be incredibly useful.
If it seems chaotic, consider the Unix PATH variable. Any time you type a name, the shell looks in lots of locations and runs the first one it finds. That's effectively the same sort of idea as autoloading. Yes, you can do some really nasty magic if you want, but then you can do that with the shell to, and generally people get along find.
Anyway, vaguely curious about it I decided to do some digging around, and came across this post by Leif K Brookes, which suggests this:

You could wrap it in an object, but that's a bit of a hack.

Europython

Michael — Wed, 10 Jun 2009 16:35:47 +0000

I've mentioned this in a couple of places, but mentioning on my blog seems appropriate too.

New Tool for helping debugging Kamaelia Systems

Michael — Mon, 08 Jun 2009 17:17:59 +0000

I've added a PythonInterpreter component to Kamaelia in Kamaelia.Experimental.PythonInterpreter. The primary use for this is intended to assist with debugging, but there are other possible uses.

Use in a system

This allows a variety of things, from a basic command line console:

StandaloneInterpreter().run()

Which you can run in the background on any console. For example you could do
this:

ServerCore(protocol=myprotocol, port=1234).activate()
StandaloneInterpreter().run()

Alternatively, you can use an embeddable component that speaks to inbox/outbox rather than stdin/stdout. Crudely you can do something like this:

Pipeline(
ConsoleReader(),
InterpreterTransformer(),
ConsoleEchoer(),
).run()

But you can also put it inside a pygame application, reading & writing from a
Textbox/TextDisplayer:
Pipeline(
Textbox(size = (800, 300), position = (100,380)),
InterpreterTransformer(),
TextDisplayer(size = (800, 300), position = (100,40)),
).run()

Going from generator coroutines to Kamaelia Components

Michael — Sun, 17 May 2009 01:10:56 +0000

Earlier this evening an announcement by Pete Fein regarding the formation of a python-concurrency mailing list (aka Python Concurrency Special Interest Group) bounced past my inbox in the unladen-swallow mailing list. Naturally, given my work on Axon (a concurrency library) and Kamaelia (a bunch of components that use it), it jumped out at me as interesting. (5 minute overview for those that don't know what Kamaelia is...)

Bar Camp Leeds UK

Michael — Tue, 05 May 2009 20:56:42 +0000

Bar Camp Leeds (UK) is running for its 3rd year running. It'd be great to see other Python people there if you're around. Details:

May 30th, May 31st
Old Broadcasting House, Leeds, UK
Website: http://barcampleeds.com/

Schedulers matter

Michael — Wed, 04 Mar 2009 10:26:54 +0000

Improving the scheduler. It's been something we've avoided for quite a while in Kamaelia, but Simon Wittber's recent post benchmarking Fibra vs Kamaelia vs stackless is interesting. His key showing that "Stackless is 7x faster than Fibra, and Fibra is 10x faster than Kamaelia" are cool for him :-) ... and naturally led me to think "why". Looking at the code, it struck me that he's doing something more interesting with the scheduler given these code forms:

scheduler.install(self.messageLoop())
# self.MessageLoop is a regular python generator
...
yield self.incrementCounter()
yield kickTo.messageQueue.push(self)

If you delve inside the fibra scheduler (which doesn't appear to be here unexpectedly) you see the following core:

    def tick(self):
        """Iterates the scheduler, running all tasks and calling all
        handlers.
        """
        active = False
        for handler in self.handlers:
            active = handler.is_waiting() or active
        active = (len(self.tasks) > 0) or active
        tasks = []
        while True:
            try:
                task, send_value = self.tasks.pop(0)
            except IndexError, e:
                break
            try:
                if isinstance(send_value, Exception):
                    r = task.throw(send_value)
                else:
                    r = task.send(send_value)
            except StopIteration, e:
                r = e
            if r is None:
                tasks.append((task, None))
            else:
                try:
                    handler = self.type_handlers[type(r)]
                except KeyError:
                    raise RuntimeError("Don't know what to do with yielded type: %s" % type(r))
                handler.handle(r, task)
        self.tasks[:] = tasks
        return active

The core of this appears to be "when I'm done, do this later" next. If you think that's familiar, it should be - its very similar (at least conceptually) to what twisted does with deferreds. It's not identical, but similar. So what does this mean for the hacksack demo? Well, if we look at self.tasks after each run, by changing:

    def run(self):
        while self.tick():
            pass

to:

    def run(self):
        while self.tick():
            print "TASKS", [x[0] for x in self.tasks]

It becomes apparent what's happening (though it's fairly obvious from above):

TASKS []
TASKS []
TASKS []
TASKS []
TASKS []
TASKS []
TASKS []
TASKS []
TASKS []
TASKS []
TASKS []
TASKS []

Fundamentally, the reason it's quicker for two reasons:

It always knows which generator is ready to run next.
It also defaults to pausing the generator, unless it specifically asks to be run. ie the tasks default to being passive.
The sender knows who the receiver is. This allows the sender to schedule the receiver explicitly.

These two points matter because Axon's scheduler (from jesse's post, hopefully people are aware that Axon is the relevant part of kamaelia here):

Is effectively a round robin scheduler - essentially for simplicity. The key question we wanted to ask was "does a component model based around inboxes/outboxes make it easier to make easier to write/reuse concurrent software", rather than "how can we build a faster, more responsive scheduler". As a result the round robin scheduler was always a compromise. There's a little bit of intelligence in there, but not much.
Kamaelia's components default to active, rather than passive. That is they're expected to run continuously unless explicitly paused. This design decision impacts the scheduler as a whole.
The sender does not know who the receiver is. This requires something different to happen in the scheduler. On the upside, it means that code can be tested easier in isolation, or reused in situations it wasn't originally expected to be used within.

This leads me to the rather obvious solution here - can we rebuild a better Axon scheduler by reusing fibra and throw away our scheduler? If so that would be really neat - throwing away our scheduler for something faster and more intelligent would be rather fun :) If we can't then we've been pondering improving it - that is making it more intelligent - for a while. Fibra's benchmarks suggest that doing so would be well worth it. The question this raises though is whether doing this would help us in the general case. At present I'm unclear on that, but until you try, you don't know.

The 5 minute (version), not the full half hour...

Michael — Thu, 26 Feb 2009 23:15:28 +0000

As you may know Kamaelia is a project I'm rather familar with (ok understatement), but one of the key ideas behind it is about how to make concurrency usable for everyday problems (from greylisting, database modelling assistance, though to learning to read & write). No one concurrency method suits everyone - as Jesse's recent series of posts shows (if you've not read them they're well worth it :), but Kamaelia's approach fits my head, and hopefully yours too. There may be a few areas where nicer syntactic sugar may be appropriate and so on, and feedback is always welcome - preferably to the project's google group/mailing list.

OK, the point of this post is that recently at O'Reilly Ignite UK North I gave a 5 minute talk which was essentially "what we've learnt about practical concurrency", and by necessity of time it's a whistle stop tour. The slides are on slideshare, and the video is on blip.tv - along with the rest of the talks - and by the magic of the internet they're below as well.

Embracing Concurrency (slides)

Embracing concurrency for fun utility and simpler code

View more presentations from kamaelian. (tags: concurrency multicore)

Embracing Concurrency (video)

Actor Model vs Denial of Service

Michael — Mon, 17 Nov 2008 03:40:21 +0000

Kamaelia implements something similar to, but not the same as the Actor Model. Since I'd not heard of the actor model for the majority of time whilst working on Kamaelia (kamaelia is more like occam, unix pipes, hardware, etc), I've been reading up on it. One thing that I've come across about it that suprises me is this:

Communications among actors occur asynchronously: that is, the sendingactor does not wait until the message is received before proceedingwith computation.
(this summary courtesy of the Wikipedia page on the Actor Model, but the literature supports this view as well)

The difference between Kamaelia and the Actor Model is in a few places, but possibly the most fundamental is this:

With the actor model (as I understand it)

You have a mailbox where you receive messages. ("recv from inbox")
communications defaults to unidirectional, though you can build bi-directional

When you send a message, you know the recipient and send it direct to the recipient. ("send to bob")
Message sending is asynchronous
ie sender knows receiver, receiver does not (necessarily) know sender

This potentially introduces coupling in ways that makes co-ordination languages harder to build

With Kamaelia's model:

You receive messages from multiple named inboxes (ala receiving data on stdin, or receiving signals)
communications defaults to unidirectional, though you can build bi-directional
You send messages to named outboxes (ala sending to stdout, stderr)
A higher level co-ordination language (effectively) joins the dots between outboxes to inboxes (generally)
Message sending defaults to asynchronous, but you can define a "pipewidth" or "max number of messages in transit" to allow synchrony, where needed (such as a producer that produces data faster than the consumer can consume it)
ie sender does NOT know receiver, receiver does not (necessarily) know sender

Much like a CPU doesn't know whether it's plugged into a motherboard or a testing harness for example.
This defaults to needing a co-ordination language - but this encourages greater reusability of components, through a dataflow model

I say "kamaelia's" model here, but this is pretty similar to hardware, unix pipes, etc.

Now the thing I can't tell by looking at literature is what the general case is for most actor systems in practice:

Do they do the literal, and solely asynchronous thing? (ie always return immediately without error when you send a message)
Or do they introduce the ability to add in synchrony where necessary? (block or return an exception when a maximum inbox size is reached)

The reason I ask is because if you don't have the ability to define pipewidths, maximum inbox sizes, maximum outbox sizes or something then you can easily cause a denial of service attack in that scenario by having producers overload consumers. Consider a frame grabber feeding a slow, experimental video codec. In that scenario, it becomes rather important to be able to have a form of blocking (or EAGAIN) behaviour available. Indeed, this is such a common issue, that it's why unix pipes & filehandles will buffer a small amount of data, but block if you send too much data (or exhibit EAGAIN behaviour :). Similarly this is what's behind the socket.listen(argument) call in a server - to allow a certain number of connections to queue up, before refusing connections...

Simple Chat Server in Kamaelia

Michael — Thu, 13 Nov 2008 11:00:14 +0000

On the #kamaelia IRC channel , the issue was raised that a number of the examples on the Kamaelia website are just that, small examples, perhaps too small to be useful beyond illustrating the basic idea. (I'm not sure this is universally true, but I accept its definitely the place in more places than we'd like!). Now, one thing we've got planned for the december release is to include a lot more examples, and as a result I'm interested in hearing what sort of examples people would find useful. For example, the example mentioned on the channel is "what does a simple chat server look like" ? (and hence why I'm interested in what examples people would find interesting/useful)

A post per day in November

Michael — Sun, 02 Nov 2008 14:39:11 +0000

Interesting new meme, I think I'll jump on this bandwagon. For my first post in this vein, I ought to retroactively do November 1st's post. So, in that spirit of catching up, I'll post the following short presentation I wrote to answer a query posted on my blog a few days ago. As it indicates, it's a walkthrough about what happens inside a particular, simplilfied layered protocol written in Kamaelia. It's probably worth noting that every "box" in the diagrams is a separate component, meaning it runs concurrently with other components.

Kamaelia Protocol Walkthrough

View SlideShare presentation or Upload your own. (tags: concurrency python)

The reason for it being that form is because it is significantly clearer than text.