Autoloading in python

Posted by Michael on 21 Jun 2009 at 16:14

Before I started using python, I'd used perl for several years, and one thing which I'd liked about perl was its autoload facility. Now in python the closest equivalent that I've seen is __getattr__ for classes, but not __getattr__ for a module. This seemed like a real shame since there are times when autoload can be incredibly useful.
If it seems chaotic, consider the Unix PATH variable. Any time you type a name, the shell looks in lots of locations and runs the first one it finds. That's effectively the same sort of idea as autoloading. Yes, you can do some really nasty magic if you want, but then you can do that with the shell to, and generally people get along find.
Anyway, vaguely curious about it I decided to do some digging around, and came across this post by Leif K Brookes, which suggests this:

You could wrap it in an object, but that's a bit of a hack.

import sys

class Foo(object):
     def __init__(self, wrapped):
         self.wrapped = wrapped

     def __getattr__(self, name):
         try:
             return getattr(self.wrapped, name)
         except AttributeError:
             return 'default'

sys.modules[__name__] = Foo(sys.modules[__name__])

That looked reasonable, so I created a file mymod.py which looks like this:

import sys

def greet(greeting="Hello World"):
   print greeting

class mymod_proxy(object):
    def __init__(self):
        super(mymod_proxy, self).__init__()
        self.wrapped = sys.modules["mymod"]
    def __getattr__(self, name):
        try:
            return getattr(self.wrapped, name)
        except AttributeError:
            def f():
                greet(name)
            return f

sys.modules["mymod"] = mymod()

And tried using it like this:

~> python
Python 2.5.1 (r251:54863, Jan 10 2008, 18:01:57)
[GCC 4.2.1 (SUSE Linux)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import mymod
>>> mymod.hello()
hello
>>> from mymod import Hello_World
>>> Hello_World()
Hello_World

And as you can see, it seems to work as expected/desired.

Now the reason I'd been thinking about this, is because I'd like to retain the hierarchy of components in Kamaelia that we have at the moment (it's useful for navigating what's where), but given we tend to use them in a similar way to Unix pipelines it's natural to want to be able to do something like:

from Kamaelia import Pipeline, ConsoleReader, ConsoleWriter
Pipeline(
ConsoleReader(),
ConsoleWriter(),
).run()

Rather than the more verbose form specifically pulling them in from particular points. Likewise, we don't really want to import every single module in Kamaelia.py, because of the large number of components there (Kamaelia is really a toolbox IMO where things get wired together, and Axon is the tool for making new tools), the majority of which won't be used in ever application!

Now, I haven't done this yet, and wouldn't do it lightly, but the fact that you can actually make autoload functionality work, seems kinda cool, and and a nice opportunity. But I'm also now wondering just how nasty this approach seems to people. After all, Leif describes it as "a bit of a hack", and whilst it's neat, I'm not taking in the positive view. I'm interested in any views on better ways of doing Autoload in python, and also whether people view it as a nice thing at all. (One person's aesthetic is another person's pain after all...)

Reply to this post

Comments

Maybe this is nicer? #

Thinking about it, maybe this is nicer?

def greet(greeting="Hello World"):
   print greeting

class autoload(object):
    def __init__(self, __name__):
        super(autoload, self).__init__()
        self.wrapped_name = __name__
        self.wrapped = sys.modules[__name__]
    def __getattr__(self, name):
        try:
            return getattr(self.wrapped, name)
        except AttributeError:
            def f():
                greet(name+" "+self.wrapped_name)
            return f

if __name__ != "__main__":
    import sys
    sys.modules[__name__] = autoload(__name__)

Doesn't hardcode the module name, and doesn't try to overwrite __main__, which seems more appropriate.

-- Michael, 21 Jun 2009 at 16:30, Rating: 0 (Reply) (Moderated by: anon)

Lazy imports #

As far as I can see, you just need to be able to do:

from Kamaelia.some.hierarchy import SomeComponent

in your main Kamaelia.__init__ module, without actually doing the import.

So, here's how I would do that:

import sys
class ObjectImportProxy(object):
    def __init__(self, module_name, object_name=None):
        self.module_name = module_name
        self.object_name = object_name

    def __getattr__(self, name):
        __import__(self.module_name)
        obj = sys.modules[self.module_name]

        if self.object_name:
            name = self.object_name + '.' + name

        for part in name.split('.'):
            obj = getattr(obj, part)

        return obj

Pipeline = ObjectImportProxy('Kamaelia.shell', 'Pipeline')
ConsoleReader = ObjectImportProxy('Kamaelia.io.read', 'ConsoleReader')
ConsoleWriter = ObjectImportProxy('Kamaelia.io.write', 'ConsoleWriter')

-- flowblok

-- guest, 22 Jun 2009 at 00:54, Rating: 0 (Reply) (Moderated by: anon)

why? #

out of curiosity, what is the (preferably real world) use case for this kind of functionality?

anyway, you can hack importlib stuff, ala http://sayspy.blogspot.com/2009/06/lazy-imports-in-25-lines-of-code.html for a more generic/implicit way (you can do this kind of thing for every module for example). DISCLAIMER: i think it would be very bad idea :-)

PS i'd recommend __getattribute__ as opposed to __getattr__, there are better chances it is called

-- guest, 22 Jun 2009 at 08:05, Rating: 0 (Reply) (Moderated by: anon)

Support for IDE #

Not to start a flamewar, since some people think Python + IDE == Evil, but: wouldn't this approach destroy any chance of autocomplete working with an IDE such as WingIDE? Of course, you could publish .pi files along with your modules.

That said, it looks like a neat idea. Lazy loading is always a good idea if you're looking at loading dozens of related, but possibly not used, modules.

This might help you on your way too:

http://code.activestate.com/recipes/473888/

Actual lazy loading, and it doesn't appear to destroy auto completion.

BTW: at the bottom of the comment page, it says: FATAL: Could not load 'dojo.widget.Editor'; last tried '__package__.js'

This is on Konqueror on Kubuntu 3.5.10

-- guest, 22 Jun 2009 at 08:23, Rating: 0 (Reply) (Moderated by: anon)

use types.ModuleType #

that's a nice trick but you should consider make your module facade object inherit from types.ModuleType to get something which looks more like a module. see for e.g.: http://alxr.usatlas.bnl.gov/lxr/source/atlas/Control/AthenaPython/python/PyAthena.py?v=head#134

-- guest, 22 Jun 2009 at 12:48, Rating: 0 (Reply) (Moderated by: anon)

Back to front page