Hacker News Reader

Comment by bob1029 | Hacker News Reader

Stack traces are your #1 ally when supporting someone else's legacy production pile.

Once you get comfortable with how they work and what information they contain, you can hit the ground running anywhere. Stack traces will teach you about the product architecture faster than anyone on the team can.

As you embrace them, you take the little bit of extra time to make sure they go well. For example, re-throwing exceptions correctly, properly awaiting results, etc. Very minor details that make all the difference.

A broader outcome of this enlightenment is preference for monolithic products. Stack traces fare poorly across web service and API boundaries. If you've only ever worked with microservice architectures, the notion of a stack trace may seem distracting.

View on HN

> A broader outcome of this enlightenment is preference for monolithic products. Stack traces fare poorly across web service and API boundaries. If you've only ever worked with microservice architectures, the notion of a stack trace may seem distracting.

Yes. People forget that the original concept of microservices, the AWS "everything must have an API", was to put in an accountability boundary across teams. Either the API behaves per its contract or it does not, you're neither expected nor really allowed to cross that boundary into the API to find out why it's doing that.

In an environment which is correctly doing "each microservice is a different small team", that helps. In an environment which is doing "one team maintains lots of microservices", this is nearly always an anti-pattern.

Accountability and transparency go hand in hand, though. Teams would be able to debug together much easier if stack traces were propagated across RPCs, and good RPC frameworks can do that. Unfortunately when (ab)using HTTP+JSON for RPCs, good cross-service debugging is one of the first casualities.

I have a hard time imagining what that means but it sounds like it could be helpful for a project at work. Can you point to a source that elaborates on error propagation on rpcs?

It means if thread A does an RPC to a remote service, and the thread B in that service that's handling the RPC takes an exception, the details are serialized and returned back to thread A where the exception is rebuilt and rethrown, ensuring that in a platform that supports chained exceptions and blocking threads efficiently (e.g. JVM) that you get exceptions and stack traces that cross services.

It's not hard to implement, it's just that the industry kinda gave up on using powerful RPC frameworks with language integration. gRPC is about the fanciest you'll get and that is a clone of Stubby, which was originally written for C++ in a codebase that banned the use of C++ exceptions entirely. So it doesn't try to solve that problem.

> the industry kinda gave up on using powerful RPC frameworks with language integration

The trouble is that ties both ends to the same implementation language. Web-style RPC became popular because its simplicity meant it wasn't tied to any particular features of any particular language.

Yes, and simple textual protocols that the browser's limitations made ubiquitous.

I hope that at some point someone standardizes a way to pass virtual stack IDs, exception data and other RPC metadata via HTTP headers so we can get back the sort of features you find in high end RPC stacks.

Do you know of any methods not tied to a particular RPC framework that attempts to solve this problem? I imagine that you could try to implement something of the sort using OpenTelemetry Tracing at least, although the need to (presumably) reconstruct the distributed stack trace from a separate store is a deficiency of this approach for sure.

Not off hand, there's some JSON problems spec I found once but I don't recall if it had anything for stack traces specifically.

It's not hard anyway. You can just upgrade server frameworks to render the stack trace to either a string or a json data structure (or protobuf), and when an exception bubbles up you capture it, convert it to that structure, compress it, possibly encrypt it with a shared key (just to avoid leaks in case there isn't a proxy stripping the special header), base64 encode it and stick it into a special header. Then your HTTP client can be taught to look for that header in case of a 500, decode it, turn it back into an exception and rethrow it.

If you have a unified framework like Micronaut it's probably the work of an afternoon to throw together. The hard part is the crypto. The moment you introduce keys, corporate security teams will insist on things like frequent rotation even if it's just a backstop against badly configured frontend servers and not mission critical (if you leak a stack trace, ok, not great but not the end of the world). So it might be better to obfuscate in a different way that isn't meant to be fully secure, or detect if a request comes from an internal IP in a secure way, etc.

Reply on news.ycombinator.com