Crossbar.io: Real-Time Distributed RPC in Python (Interview)

Django channel is a way to mitigate django limitations. In that sense it does work but it doesn't do a 10th of what crossbar do.

Channels let you do websocket, but crossbar add a standardizes a protocol on top of it for rpc and pub sub. It propagate errors for you, allow wild cards in routing, load balancing for rpc, and allow php, c#, js and python code to talk to each others transparently. But the best feature is: crossbar route all messages for you and clients don't need to know about other clients, only about the crossbar. It takes a time to realize how important this is, but it changes the whole architecture of your app.

Channels are basically only non blocking data pipes for django, with one for websocket. Useful, but nowhere as powerful.

I've made a living with django for years and love the framework, but channels feel like a hack to me. I hope for a django 2.0 with backed in async/await.

Authentication is working quite well but involves a lot of manual work right now. Ex: for django you need to create an auth microservice calling django.contrib.auth authenticate function.

Now as for pub/sub, I use uuid to namespace messages and rpc call to fetch the namespace if i want to limit their access. It's imperfect but you have the same issue with redis or bare socket.js.

However, last year tavendo added meta events and rpc allowing you to react to sub and control them from the api, so you can roll the policy you want. Again, manual work. Hence the framework I wish we had.

The thing is, tavendo targets IoT with crossbar and is developping it's acticity for the factories and office automation. If we want a mature web solution, somebody will have to code it.

You are right. It took me quite some time to get there and I did it only because I was looking exactly for something like this.

My advice would be to start with only the js part in the browser, at it's the easiest piece to understand. You can use the demo crossbar instances to avoid installing it.

This is why this tech need a framework on top of it. Right now it's quite bare bone. A lot like nodejs when it started.

Django doesn't have "the one part". It's doing A LOT:

- HTTP response life cycle

- templating

- routing

- database access layer, including migrations

- admin generation

- authentication

- session management

- project life cycle

- project layout

- serialization

- caching at several layers

- settings management

- web app sdk (with reusable views and pluggable apps)

- data validation and form handling

- event handling (mainly using signals)

- security management (csrf token, xss escape, sql escape, signed cookie, ALLOWED_HOST...)

- testing framework

- dev toolkit (autoreloading runserver, auto server static file, etc)

- command line management

- i18n and l18n (including text translation and timezone handling)

- RSS feed generation

Django is HUGE. It is great because despite being huge, it's very, very, very well integrated.

This is why pyramid didn't catch on as much: it's as big, but not as well integrated.

This is why flask get away with it: it's not as well integrated, but it's not big.

If you want to handle complex technologies like the one brought by crossbar, you can't expect most people to do it by hand. Not only it's not productive, especially at the begining of the project, but most devs simply don't have the skills to do so, nor should they have to.

Imagine if most people had to do the CSRF code in Django. Or the create the admin by hand. Or reroll their authentication method every time. Do you think they would do it right ?

Two months ago I worked with some administration, and they had a flask site. Because flask doesn't come with auth, instead of looking for a flask blueprint with it, they rolled their own. I was based on md5 hashed passwords without a salt. I kid you not.

> Django integration sounds interesting and it's something we have been mulling about. Could you point any specific examples or code? How did you deal with RPC and Pub/Sub with Django? Solutions that come to mind are adjacent view functions that Crossbar calls when messages arrive (meaning users wont touch those directly) or perhaps going directly to Django with scripts (Django management commands)?

There are a lot of things to get Django integration because Django is a lot of things, but I can see 3 points worth mentioning quickly:

- auth. Crossbar has it's own version of auth, which means you need to create a WAMP auth service and call django.auth.contrib's authenticate() function in it to get authentification in it. It also means you need to create some manual hook for logging out.

- pub/sub and rpc: crossbar has an HTTP bridge, and it's probably the best way to get Django in the loop for all events. However, you must remember than you may have many Django processes in production, and only one will receive each pub.

- orm. Django's ORM is blocking, meaning you have to run all calls to it into loop.run_in_executor. It's very annoying and there is no way around it.

> I understand you'd build these features on top of, or relying on crossbar.io?

Yes. Although the webpart would be handled by something else, but connected with an integrated wrapper.

> This sounds like something regular Django wouldn't have a place in.

Indeed. Django being blocking, you can't really make it the center piece of such a work. You could, however, provide adapters so you can integrate with existing Django project to ease transitions of existing code bases.

> I'd very much like a system with features these features.

Maybe I should start a kickstarter or something like, gather a team and work on it. I gave it a try in short sprints, but a few hours does not cut it. After a while I ended up opening several tickets on the official asyncio repository because as soon as you work that close to the event loop, you run into edge cases you must take in consideration.

> Django's structure and direction doesn't seem to allow things like this. Class based configuration for various things, use of startup time settings everywhere. Django has a very wide and stable base for regular web stuff, but when you need things like JS web apps, streaming, reactivity, asynchronic distributed operations it gets complicated pretty fast. After more than 10 years with Django I'm still unsure on how to tackle things like this properly, without hacks and loads of boilerplate.

We can't blame Django for that though. It has been a useful design for a long time, plus the design is still holding quite well today. It's just starting to show limitions because we compare it to much more recent techs. Still, I agree.

AppConfig and Django channels are hacks written as a workaround of the framework design.

But let's not kid ourself, the amount of work that went into Django is enormous, and it's still the framework making me the most productive to this day. Any alternative would have a hard time just catching up with what it can do before even starting to offer killer features that Django doesn't have.

Some other issues:

- async is hard. A good async framework should aim at making common task as easy as possible, including debugging. This is really hard, and few developpers spend time on that.

- async is hard (again). A good async framework should aim at providing a solid even management framework, with a clear lifecycle and robusts hooks. This is really hard, and few developpers spend time on that.

- configuration is un unsolved problem in Python. There is no such thing as a "standard configuration framework" and we all end up parsing config files, reinventing live settings in database, or dealing with redis/etcd. But configuration is a really, really hard problem : you have serialisation, access rights, value sharing, data validation, hierarchy, handling of different data source and several layers of conf depending of where in the life cycle you are in an app. This is a project on it's own.

- crossbar code base is not made to be integrated. It's a design decision from Tavendo's team: provide a piece of infrastructure (like nginx, redis, etc) and not a library or a framework. So you would probably have to make some heavy mocking and monkey patching to bend it to a programmable API. Or, god save us, making a fork.