Hacker News Reader

Comment by osener | Hacker News Reader

I like the end result of OpenTelemetry tracing when using Axiom and the like, but the SDKs have been a nightmare. Too much emphasis on automatic instrumentation, Java-isms, everything is stateful and abstracted away.

It can do distributed tracing of otherwise traditional long running microservices, but breaks down when your functions are distributed like in durable execution engines, Cloudflare Workflows, “functions” that span hours/days/weeks and steps that retry many times.

I had to reverse engineer how SDKs work and how tracing UIs display data so I could make simpler functions that fit wider variety of runtimes and more freely parent spans, start spans and end them from different function instances.

I think most of the API and terminology complexity is self inflicted. Would love to see a rebooted developer experience that is less Kubernates-brained.

View on HN

The article assumes the issue with OTel is slow feature development, which isn't my experience at all. The issue I've had is that the SDKs have terrible performance overhead for instrumentation and are, as you say, highly resistant to integrating the output of better performing (or just preexisting) instrumentation. In Python and Ruby, at least, the CPU cost of all the mandatory abstraction is way too high.

What I find confusing about this is that otel is two things.

1. A spec 2. A ref implementation

Similar to other projects (e.g. python), if there's complaints about (2), that should trigger an ecosystem of alternative implementations that are guaranteed to be compatible because of (1).

I suspect there's actually quite a few private, separate otel implementations. Maybe these just aren't being contributed as oss?

Yeah, they spent a ton of effort trying to cram automatic-config-and-library-discovery-like features everywhere when they would've been MUCH better served by requiring explicit dependency injection... and then just adding DI wrappers externally. That's what contrib is for.

As it stands, due to the tower of abstractions that could've just been "init with an implementation of this interface", you need to learn several pieces and how they work together (hint: convoluted and horrifically inefficiently) to modify any piece, and inevitably you learn that to get what you want, you need to swap out a major portion of it... but doing that while maintaining the auto-registry nonsense is a gigantic effort. If it's even possible.

It is the new poster-child for "design by committee". It's horrific. Unfortunately it's also usually the best option in large setups. I greatly approve of the high level goal, but omfg

It's crazy, IMO, that they didn't simply design otel clients. Making this giant cross language framework is an insane endeavor that just makes everyone unhappy.

Yeah, cross language is generally considered "a protocol", and exists only to cross process boundaries. That's definitely useful! It's even a mostly reasonable one (though with a few weird decisions either due to blindly copying Prometheus' flaws or due to... idk avoiding copying Prometheus on principle? Very strange sometimes, but livable). We needed a grand unification here, even if mediocre, and the time was right.

Trying to make all the supported languages feel similar (beyond sharing concepts which almost directly match the protocol) is foolish in the extreme, and it's why it's such a monstrosity. And worse, they seem to treat that as more important than the bottom-most clients that speak the protocol, so you might be waiting years for any support for a third of the system!

Even just basic wire protocol is ass that's PITA to parse, like list of attibutes (which have to be unique) isn't a map but array of maps with some weird way to encode key and type. The whole project is industrial scale mediocrity

Totally agree. However I am hopeful. We started the first full instrumented project a few years back. It took us a long time to do the whole work including understanding the SDK, mapping the dimensions and getting everything right. Our last project we did the whole thing with agents and they really took away a lot of the pain from the implementation part. We also use Axiom MCP so when we need some trace or event in the logs the agents look for it and if they don’t find it they’ll add it for the next time. It’s really been a different experience.

I tried to emit metrics from a python app using otel once. Gave up and switched to prometheus. What a nightmare.

OpenTelemetry reminds me a lot of the bad old days when Java/XML maximalism was fashionable.

Ran into the same issue and didn't find any willingness in the OTEL gods to close this gap.

The official Go SDK is a nightmare to use and understand. Just the "getting started"[1] requires adding 10 imports.

[1]: https://opentelemetry.io/docs/languages/go/getting-started/#...

Reply on news.ycombinator.com