Comment by jamesdutc | Hacker News Reader

Comment by jamesdutc | Hacker News Reader

The mechanism is called "import hooks":

https://docs.python.org/3/reference/import.html#import-hooks

https://www.python.org/dev/peps/pep-0302/

You can see the import hook added here:

https://github.com/kragniz/json-sempai/blob/master/jsonsempa...

Import hooks are a great feature in Python.

Take the `import` mechanism in the language and reduce it to its barest theoretical formulation - what does it really do? (Name binding.) Think of all the other features in your application that can be reduced to this. (e.g., configuration management) Use the constraints of the import mechanism to guide the design of this feature, and use import hooks to implement it. You'll end up with an implementation that is very "close" to the core language. I would assert that this closeness strongly suggests correctness and composability.

Less philosophically, think of the Python virtual machine as a system with lots of "safety valves" and "escape hatches." Import hooks are one such safety valve. Think about all the things you could do easily by hooking into module importing? (Real use-cases: loading code from bitemporal object databases, deprecation of modules, relocation of modules, configuration management, &c.)

Of course, there are languages that are much more flexible than Python in this respect. Python aims for a practical flexibility, and I find that, in practice, Python strikes a nice balance.

Import hooks are awesome, one of the cool things you can do is re-write the code as you load it. I made a toy loader that automagically inlines (some) Python function calls[1]. There are also cool projects like MacroPy[2] that do much more extensive things.

1. https://tomforb.es/automatically-inline-python-function-call...

2. https://github.com/lihaoyi/macropy

Awesome, I saw your post a while back and it triggered me to write a full inliner for python, also added some IFDEF macros using context managers e.g.

with IFDEF("DEBUG"):

I'll have to dig around and find my code, your inline project was a great help in understanding the import hooks and walking the ast.

I read your experiment on inlining and it looks quite interesting. Did you do a benchmark on how it affects (improves?) performance? In my line of work I've found ocasionally places where inlining would've helped (functions that do masking with 64 bit masks, for example, are a mess to "inline by hand" and kill a lot of readability and clarity). Even with the current limitations of your "toy" implementation, it seems like it would help to avoid the costly function calls.

Sounds a bit like the Ruby-fication of Python.

A simple example of how import hooks use for module relocation/deprecation:

https://gist.github.com/dutc/0f7498451d98e3114268

This seems like a huge case of "implicit over explicit". Ideally if you import package foo, it shouldn't affect anything but foo.bar

Python itself never enforced that, though. It's the libraries' responsibility to have a "register_import_hook()" function or similar instead of just doing so unannounced.

For what it's worth - modern JavaScript does it too, it separates the syntax (import/export) and semantics of binding the reference from the actual loading mechanism. Not only can you "hook" into imports, you can replace them altogether if you'd like.

The "loader protocol" (PEP 302) is a related trick that can be used locally, so it only affects imports from packages that opt in. I use this in https://github.com/bdarnell/codegenloader to automagically process protobuf/thrift files and import the generated code.

Reply on news.ycombinator.com