Hacker News Reader

Comment by BackBlast | Hacker News Reader

What is "correct tooling"? I haven't found anything that's remotely close to providing a nice stack trace, for example. How "micro" are your services? Can it manage to stand up a local dev environment? How do you deal with service interface versioning? Is this great tooling vendor tied?

View on HN

On the stack trace, I think this is what the modern "observability" stuff is all about; traces, wide events, etc. One event per request. DataDog will say they do this, Honeycomb will say they do it and that DataDog is kinda lying, now there's OpenTelemetry, it's a deep rabbit hole.

It's easy to say this is a lot of work to reinvent something you get for free with a (single language) monolith, but at least it's recognized as a problem worth solving.

The stack trace bit is hard. For local development, ideally there's some fancy service discovery where your local service can hit a development instance of another service if you don't have a version running locally.

Given sufficiently carefully designed logging (with a request id that starts at the outermost service and gets propagated all the way through) you should be able to see the equivalent in the logs from a development set of services when something goes wrong. Pulling a full request's logs out of production logging is a bit trickier.

For me, it boils down to "this is absolutely doable but I'd still rather have as few services as possible while still maintaining useful levels of separation" - at least for the primary business services. Having a bunch of microservice like things serving pure infrastructure roles can be much cooler depending on your situation.

I agree that such a system could be designed and built. I just haven't seen any tooling that provides it, the next thing to free, like modern languages do. As far as I can tell, you have to have an experienced developer craft the system such that these features work. They don't come out of the box with any tool set I've seen, and I'm still looking and asking.

Lightstep is pretty cool and will show you something like a stack trace across systems, with timings, but it's not for free (monetarily, or dev cost to integrate it into your stack)

Yes, I see. I tried to understand their pricing. 10,000 active time series? Does that mean it will store and let me view 10k top level API calls for their fee? 10k separate services? I don't quite understand how this maps to actual use.

$100/service. Is that top level service? To cover a 100 endpoints at one service each is going to be 10k/mo?

I believe the 10k time series is how many complete end to end traces (across multiple services) it will store at once. Lightstep has settings where you set sample rate, retention, which traces you want to collect etc.

I believe $100/service is top level service, not endpoint. But really not sure.

Reply on news.ycombinator.com