Hacker News Reader

Comment by lfmunoz4 | Hacker News Reader

Apparently it is a spring boot app. Why are kafka and redis dependencies? Also I see Netty is used but not sure how was the MQTT protocol implemented, which library was used or was implemented in java from scratch?

Also why wasn't emqx used? Why is this solution better for their use case?

View on HN

You are correct. TBMQ is based on Spring Boot. We use Netty (https://netty.io/) as the source of the MQTT communication, and we build the MQTT features the MQTT broker should support ourselves on top of that. Kafka is used to implement scalable, fault-tolerant solution with reliable message persistence. Additionally, it helped to gain high-throughput processing and low-latency message delivery. We store published MQTT messages, client sessions, and subscriptions, etc., in Kafka. It proved its quality in practice and with performance tests. Redis is currently used for caching in cluster mode. In our roadmap, we also have a ticket to add the ability to store MQTT messages there. So, it will be a so-called Redis integration.

Please see the architecture doc for more details: https://thingsboard.io/docs/mqtt-broker/architecture/

We have implemented TBMQ to satisfy our enterprise customers' needs. Some of them used hivemq before but decided to ask us to build a new solution (TBMQ) as they were not fully satisfied with hivemq. One of the most important advantages we provided is cluster support in our open-source version compared to hivemq.

emqx presumably wasn't used because they are a Java shop and emqx is written in Erlang. Apparently they have a need for a massive scaling MQTT broker. At that point you need to be seriously comfortable with the tools you are using. Whether that's Java, Rust, Erlang, C#, C++ or whatever. Using an existing application in a language/ecosystem your team is not experts in is probably going to be a problem unless you seriously invest in the new language/ecosystem first. And it's going to take at least something like a thousand hours to really become good at something. The other option is writing an alternative in the language you are already familiar with. So in that regard I can understand why they did it.

I guess they could have used the OSS HiveMQ version as a base though. No idea why they didn't do that.

Exactly, you described it well. I have answered above for the reasons why HiveMQ was not the final choice.

I'm yet to see a project where Kafka was used for anything other than resume building. My guess would've been that this one is no different.

Also, sometimes it could be used because in the previous company where the project author worked they've used Kafka. And the cycle continues.

We do not agree, but everyone has an opinion we should respect :)

Nah, you don't need to respect cynicism.

Reply on news.ycombinator.com