Is there yet any way of having a front end for this which doesn't significantly dent your revenue stream either in staffing, infrastructure or license fees? We land over 2000 requests/second and it's expensive just keeping logs.
For metrics and logs, sampling isn't so useful, so I don't have a good answer. Datadog has 80% gross margin, so at most 20% of what you pay them is the infra, so you stand to save a lot of money running your own open source stacks if your labor costs would be less than that 80%. With datadog, we have a project every 3 months to reduce usage, so it's not like we aren't constantly babysitting it anyway.
FWIW we sample in datadog APM and it works fine to control costs, I'm not sure what issues you hit.
In my experience, Datadog's ingestion sampling works pretty well.
And there's retention filters you can use to override.
Cost is one thing but you would be surprised how heavy observability can be on a service, it's uses a lot of %cpu.
The former use case is often solved by a specific and narrow kind of observability data, which is metrics. A common tool for that purpose is Prometheus. You certainly can't query Prometheus for individual requests, which is fine, and accepting that invariant allows Prometheus to treat input data as statistical, in the sense that you mean in your comment.
But if we're talking about general-purpose telemetry, we're talking about more than just high-level summaries of system behavior, we also need to be able to inspect individual log events, trace spans, etc. If a user made a request an hour ago with reqid 123, I expect to be able to query my telemetry system for reqid 123 and see all of the metadata related to that request.
A telemetry system that samples prior to ingest certainly delivers value, but it can only ever solve the first use case, and never the second.
Traces came on the order of hundreds/second, but we didn't have downsampling turned on, just collected all. Traces were saved 7 days (configurable). Very little actual optimization at the point where I left.
I think it cost on the order of dozens to hundreds of dollars a month.
There's an environment variable which you can set on your containers which defines how the tracing sampler behaves. It's in the docs. See OTEL_TRACES_SAMPLER
https://opentelemetry.io/docs/specs/otel/configuration/sdk-e...
FWIW I now use Tempo because I have everything else in Grafana (Prometheus, Loki), but I do miss using Jaeger.
Tempo can be spun up with docker compose using a local disk for ephemeral storage/querying: https://github.com/grafana/tempo/blob/main/example/docker-co...
Maybe this meets your needs?
> Jaeger is easier to setup/manage and has a better interface than Grafana/Tempo
What do you enjoy about the Jaeger interface? Perhaps it's a gap in Tempo we can improve.
I'm fairly sure there was an official Grafana-provided Jaeger gRPC plugin for Tempo, but can't easily find it, only this one: https://github.com/flitnetics/jaeger-tempo
Typically seeing a 0.1 compression ratio on data before other optimizations.
I have it connected to Fly.io here: https://scratchdb.com/blog/fly-logs-to-clickhouse/
I'd be really grateful to learn more about what you're looking for (how are you even managing logs today?) Even if you end up not using scratchdb it'll help me figure out the next thing to build!
The high-level view and being able to draw a box around something that looks weird on my graph, and honeycomb tells me what's different inside and outside the box is amazing (called bubble up, if you're searching).
It's faster than any other tool I've used; usually data is available by the time I've switched to their ui from the curl command to our API. It's mind blowing actually.
Other saas and self hosted options I've tried have all been awful in some way or other; honeycomb is a breath of fresh air, and going back to other tools after using honeycomb is painful.
I'm not sponsored or working for them btw, they just make one of the few products that I genuinely love using.
The hypothesis being that you can save money by turning things down, but easily turn them back up when you're actively investigating. Or turn the volume up for a targeted sub-segment of your traffic.
We've done some exploration into providing the same for APM and the rest of OTEL and I think it's pretty doable. hmu if you want to talk.
Could Prometheus be augmented to store metrics, logs, and traces somehow? I don't really mind if it doesn't scale well on a single instance or is highly available, I'll just add more instances and aggregate them.
This TF project does most of the heavy lift. https://github.com/telia-oss/terraform-aws-jaeger
It's in Rust to add some HN catnip.
For traces, as mentioned elsewhere sampling is key. Some systems use % of requests, and other systems cap traces to traces/sec.
Logs are just expensive but easy to use, so the best strategy is to lean on filtering to avoid a bunch of pointless repetition which don't add value.
TLDR, metrics require more planning but can end up being much cheaper than logs.