I’ve done a lot of work with Kafka, but it’s pretty heavy and I really just want a decent pub/sub queue setup between services that would be somewhat reliable.
Is there anything you’d recommend? Open to thoughts on just using Postgres as a queue as well.
At the other end of the spectrum: just use a managed queue from your preferred big cloud provider. All are reliable enough for this use case, and the cost for most projects would be zero or negligible.
[1]: https://zeromq.org/
Atm it's all in-memory, afaik. There are go & node.js implementations, which I hope some-day we add persistence to.
I'm still working on k8s Persistent Volumes, but in terms of bigger/better software, I hope soon-ish to have a Pulsar Operator[2] going. Kafka is very very exacting & specific, inflexible, but Pulsar's architecture seems much more general & flexible, capable of things like great geo-distribution by virtue of it's underlying Bookkeeper storage layer. It'd be my production go-to for any pubsub details.
[1] https://github.com/patchbay-pub https://patchbay.pub/ https://news.ycombinator.com/item?id=21639066 (362 points, 3 years ago, 79 comments)
[2] https://docs.streamnative.io/operators/pulsar-operator/pulsa...
the problem is just Kafka being heavy?
The options are many. It doesn't really matter, just do whatever will meet your needs!
When it's a personal project, I suggest settling for "good enough".
Unless you want to play with queues... otherwise I would use the database, and add redis or rabbitmq later.