Lahja – A generic event bus implementation written in Python | Hacker News Reader

Lahja – A generic event bus implementation written in Python

github.com

148 pointscburgdorf72 comments

I've been working on something similar for a while now called Lightbus: https://lightbus.org

Like Lahja this too is rather much in the alpha stage, but I have been using it in production for over a year now.

I mention it here in case it is interesting to anyone, interest/feedback always welcome!

From glancing at the code, this is an observer pattern over asyncio and unix sockets. That means you're unable to communicate outside the current machine (short of hacks like ssh socket forwarding), but adding support for tcp endpoints shouldn't be too hard.

I'd be interested to see how well this holds up under load, especially compared to other common event bus patterns.

We've talked a bit about adding TCP support and really like some of the options it opens up. However, we have to move away from the use of pickle for event serialization before we can do that, otherwise we would have some security issues with remote code execution.

Any chance the name is a play on words from the Estonian word "Lahja" as in light-weight?

Good question, could also be the Finnish word "lahja" that translates to "gift"

Could also mean 'talent'.

"lahja" in Arabic means dialect

It's also a pretty popular girl's name ;)

Also means "weak" (in concentration) in Estonian though

“Lah ja” in Hindi means “take away”

That sounds more Punjabi. It is "le ja" in Hindi.

Christoph, founder of the project here. I recently came across the Estonian meaning and jokingly used it as an explanation for the project name when I gave a presentation. However, the real story is that Lahja is the name of my daughter and I selfishly named this project after her :)

What is the closest (conceptual) analogue for this model:

- https://github.com/jodal/pykka (Python)

- https://github.com/quantmind/pulsar (Python)

- https://github.com/akka/akka (Java)

- https://github.com/eclipse-vertx/vert.x (Java)

- etc.

Neat.

This feels quite a bit like Vert.x from the Java world, which flows data between components via an event bus.

Noob python question: does Python support reactive extensions and chaining? That would fit well on top of this too.

There is RxPY: https://rxpy.readthedocs.io/en/latest/

For push-type events there is eventkit: https://github.com/erdewit/eventkit (I'm the author).

Pair this with Arrow IPC and you're golden https://arrow.apache.org/docs/python/ipc.html

That’s pretty neat!

I’ve used Redis in the past for lightweight pub/sub and zeromq for IPC before we had adopted Python 3 (and I left that job before then). I hated juggling threads and queues before that and then having to run Redis just to run your program was a pain.

The syntax looks similar enough to both of those to get started quickly and easily.

I’ve been writing Go so much I don’t know how much this is “needed” in Python but the examples using context managers ring the “it’s Pythonic” bell.

How does this compare to D-Bus and when would you use which (apart from D-Bus obviously being Linux-only)?

Is this somehow connected to processing cryptocurrencies? This is under ethereum repo.

It's a library we use in the "Trinity" client for the ethereum network. The lahja library itself is however just an event bus and has no cryptocurrency ties other than being used in the Trinity codebase.

https://trinity.ethereum.org/

what is the usecase for something like this versus celery/rq ?

AFAIK, celery doesn't provide a message passing bus between main process and sub-processes.

Well, it technically does (through its invocations of billiard, it's multiprocessing fork), but it's pretty ad hoc and not exposed for much user control. So you're basically right, I'm just being pedantic.

Yes ok initial parameter passing and gathering results may be considered a message bus. I was going to mention that, but I got lazy.

Don't understand that clearly. Can't I put something in the queue in the main process and read it from subprocesses ?

Is that going to provide bi-directional communication between main and subprocesses while they are running? If it does, disregard what I wrote.

Sure. You can put results in an "output" queue (which is backed by Redis) that is read by the main application.

Incidentally this is how you do a notification on your UI .

Having developed a similar library, I have written a (rough) response to this question here: https://lightbus.org/explanation/lightbus-vs-celery/

I suspect the answer will also apply to something like Lahja.

cannot make sense of this library. whats this for?

Seems to be pubsub for asyncio.

Love to try this out some day, a shame that this isn't in Python 2.

EDIT: Thanks HN, try working in an industry where every new project is in Python 2, and have no intention to move on, before saying "moeve to 3", it doesn't help at all.

Please, please don't use Python 2.

Tell that to the industry I work in (Games, Movies & VFX) where most of the tools & 3D software scripting are still in Python 2.

Not all of us move fast and break things here, unlike the web dev world.

Oh please. The EOL on python2 has been massively pre announced. Games, Movies and VFX can afford the port costs and have a huge liability looming.

There was no "move fast and break things" in this python 3

> Oh please. The EOL on python2 has been massively pre announced.

Nope, not when new versions of software comes out with Python 2 still, (can't do anything about those licences bought) small studios cannot afford moving/rewriting tools in P2 to 3, not a priority, rather keep P2.

Then complain to the companies you license software from for continuing to build on a soft-deprecated platform. Don't complain to developers for ignoring a platform whose deprecation was announced over a decade ago.

And certainly don't badmouth people who are using the not-deprecated tool as being ok with unstable infra.

> Then complain to the companies you license software from for continuing to build on a soft-deprecated platform.

You think we haven't been doing this? We had a representative from one of these vendors come in to discuss their roadmap, they told us straight up that "We still intend to keep using Python 2".

> Don't complain to developers for ignoring a platform whose deprecation was announced over a decade ago.

Yes, because web dev is the hot tech stuff now, nobody wants to work in a boring sector, so they are not willing to change anything, cycle continues.

> And certainly don't badmouth people who are using the not-deprecated tool as being ok with unstable infra.

I read this sentence twice and I still don't understand what you're trying to say, clarify?

> Yes, because web dev is the hot tech stuff now, nobody wants to work in a boring sector, so they are not willing to change anything, cycle continues.

Change what? You're asking for...what exactly? People to provide support for a deprecated platform? That just encourages those companies you're complaining about to continue to use python2 and further splits the ecosystem. No!

> Change what? You're asking for...what exactly? People to provide support for a deprecated platform?

Instead of trying to put words in my mouth and being hostile, try to address my concerns. What I mean is that nobody that is a dev wants to work in a slow boring sector, Which means no work gets done to Python 3 on the vendor side hence, studios like ours keep developing on it.

Nothing changes and no choice other than leaving.

Can you name some of this software?

>Oh please. The EOL on python2 has been massively pre announced.

Which is neither here, nor there. Pre-announcing doesn't give any reason or motive (or funding) for porting large codebases. Many businesses stuck with 2.x will fund 2.x maintenance if the core devs don't do it, and port on their own schedule not on whatever was "pre announced".

> Not all of us move fast and break things here, unlike the web dev world.

Then how many more years do they need to switch to Python3?

> Then how many more years

Could almost really say “how many decades” at this point.

What if they don't want to switch to anything they aren't absolutely forced to, and otherwise 2.x works for them, period?

How many years does it take companies to switch from COBOL (which still runs billions of lines of 40+ years critical code)?

> Not all of us move fast and break things here, unlike the web dev world.

I had to roll my eyes at that one. Fine if your industry is accepting the high risk of using deprecated releases for new software, but don't try to act like upgrading from deprecated software is "moving fast and breaking things". This deprecation has been announced for 6 years. Python 3 was released over a decade ago.

This ain't a new JavaScript built tool -- this is a carefully planned end-of-life situation for a very old piece of technology.

Another option is of course to use tools that take backward compatibility seriously.

Sure, do that. I'm not a Python fanboy, and there were some issues with how the transition was handled. Move to another language if you have the manpower. I think in in this case, since they claim to not even have the manpower to do some relatively straight-forward migrations from Python 2 -> 3 and/or they have some dependencies on Python2, I'm doubtful that's even an option.

I'm just pointing out that it's a risky move to adopt a soon-to-be-deprecated programming language for new projects, given the potential for security issues that won't be patched. It's the kind of ignorant arrogance that has always bugged me about many companies with non-technical management (please note: these words are not directed at user kgrave given that this appears an edict from above at his company, and he's unable to change it).

Please don't use Python 2 for any software that anyone else will ever have to work with or maintain.

Python 2 is very close to being not supported anymore, which means no more security updates, etc.

Totally agree, but unfortunately, when you're in a slow moving industry you don't have the luxury of doing rewrites on a whim.

You'd be surprised how Python 2 is _still_ being used in this day and age.

OK, but please just tell your bosses that their poor decisions to build new software in Python 2 will put your companies at risk of security exploits for the lifetime of the new software you're building in Python 2.

I can almost understand if you have legacy software that your bosses don't want to move to Python 3, but actually building new software in Python 2? It's just crazy and asking for serious problems in the very near future.

> ...But actually building _new_ software in Python 2? It's just crazy and asking for serious problems in the very near future.

Yes, sadly this still happens.

Me too but it's not been released in ALGOL 68 either!

Upgrade. Or accept that you are using a dead language.

This library uses asyncio, which is Python3. What do you expect?

You may be okay to stick with Python 2, but this means you also agree not to expect new features and new libraries. So please don't mention Python 2 to a new library.

Does your industry still use Windows XP/Vista, or PHP4, or Ruby 1.8.7, or Apache 1.x, or MySQL 4.1 or RHEL 5? Those are roughly equivalent era.

In an industry where win95 appears to be the norm (if not VAX!) but not complaining when don't see flashy libraries or tools support. It does feel good , when some claim support however.

Python 2.7 is still maintained, and the last version came out in March IIRC, so there's that.

Lots of big players still use Python 2 (Google being the most prominent), and untold smaller players.

Besides, Windows XP and PHP are also in use, as are 50+ year old COBOL systems, so there's that.

Unless you come in and do the port yourself, many systems are not getting ported anytime soon. RedHat and co will continue to release 2.7 patches for the foreseeable future, long after 2020.

Surely the number of eyes looking at that codebase will vastly diminish though? You may get RH and co backporting massively obvious issues but there will be many other smaller issues that won't be found.

Also Google have golang, which I'd imagine would be their target now, not python 3 (for migrations)

>Surely the number of eyes looking at that codebase will vastly diminish though?

I don't think the "many eyes" theory ever panned out. There are projects used by billions, and on which thousands of other FOSS projects work, and only have like a handful of people working at the codebase (and less catching bugs)

>Also Google have golang, which I'd imagine would be their target now, not python 3 (for migrations)

Not really, Golang was never Google's official/mandated language. Just a language some Googlers made with Google's use cases in mind. Google still uses C++, Java, and Python (plus some Dart). Some things use Golang, but not in some top-down "Googlers shall use Golang" way.

Where exactly does Google use python2.7?

One thing I know of off the top of my head (because I had to deal with it last week) is the gcloud suite of tools for GCP, which will only run with 2.7. Thankfully the various GCP libraries for interacting with the platform do work on Python 3.

Internally, all around. A 2018 answer by a Google engineer:

Build system automation of all kinds (although the main build tool is in Java)

Test automation

Deployment automation

Many services configure their job placement and options with a Python-derived language (not mine)

Monitoring configuration and dashboards

Data analysis using Scipy, Scikit, Numpy and various kinds of notebook UIs

Configuring Tensorflow and analysing the results of ML projects

Hundreds of little command-line utilities for all sorts of small code and data manipulations

Thousands of internal websites

The youtube.com site was still Python last time I looked

https://pythonclock.org/ - 5 Months and 2 days to go. Do not do any new projects in Python 2. There is no technical reason for you to do so. If management is an issue, show them the facts.

Also, Python 3.9 is now out which means there's been more than enough time to migrate - https://docs.python.org/3.9/whatsnew/3.9.html

3.8 won't be released until late October (https://www.python.org/dev/peps/pep-0569/). 3.9 is a ways off.

Sorry, should have stated it's a pre-release.

Seriously though, why would a company want to start a months long project of moving from python 2 to 3, when their current setup works perfectly, and the project involves the horrifying boring chore of going through every single line of code and making simple changes, and hope that their tests still works, there are no edge cases that crop up due to minor changes in the new version, etc., etc., etc. Seems much more sensible to stick with something that works.

Well, because you're developing against an unsupported runtime. No bug or security updates. If that's fine for you, feel free, otherwise get with the (supported) program.

If it's a new project, there are no excuses imho

In addition I don't think you'd have to interrogate every line, depending on your codebase and upstream module support for Python 3. There are tools like:

http://python-future.org/automatic_conversion.html and https://python-modernize.readthedocs.io/en/latest/

Which should get you most of the way there. Look at https://docs.python.org/3/howto/pyporting.html for more info

I would argue I am coding against a stable runtime. No unexpected bugs due to upgrading. No need to worry about upgrading. For many applications, I don't have to worry about security updates. I think for many cases like closed systems that plan to run for years to decades, python 2 is attractive to even new projects.

The question at this point is: why haven’t you migrated yet?

Any new code written in the last 5 years should have been Python 3 compatible (and tested).

If you’re on Python 2 you’re on a dying platform. If your code is important plan accordingly... something many have failed to do.

What industry is it, might I ask (so I can know to avoid it)?

If you say airline...

Sorry you’re getting downvoted! It’s amazing how far behind the tools in your industry are, despite Python being the standard tooling language. Proprietary tools are the worst offenders (compared with Blender has been Python 3 only since version 2.5)

For anyone interested, this is the VFX industry reference platform, which is finally hoping to move to Python 3 next year: https://vfxplatform.com/

Reply on news.ycombinator.com