What I like using Grafana Loki for (and where I avoid it) | Hacker News Reader

What I like using Grafana Loki for (and where I avoid it)

utcc.utoronto.ca

30 pointszdw14 comments

Would be interesting to hear if qryn (née cloki - clickhouse Loki) with vector might be a valid alternative to both Loki and traditional syslog.

AFAIK logshipping with journald is bespoke - but vector should be able to handle both syslog and journald - for relatively painless setup ...

Not sure if tailing is better with qryn though.

https://github.com/metrico/qryn

https://vector.dev/docs/reference/configuration/sources/sysl...

https://vector.dev/docs/reference/configuration/sources/jour...

https://www.digitalocean.com/community/tutorials/how-to-cent...

Oh interesting. I like that qryn exposes Grafana compatible query APIs. I’ve been playing with SigNoz (another Clickhouse-based, OTel compatible project for ingesting and analyzing logs, metrics, and traces) and I wish it would offer the same capability.

Thanks for the feedback! qryn developer here, and that's exactly our design goal: requiring no custom agents or clients in order to use our solution, through API emulation. There's a chance you could use qryn as a reader for data written by other clients, and that's exactly what we do with opentelemetry where we have a qryn-otel-collector allowing direct writes into ClickHouse at max speed, just like signoz, and reading through our Grafana compatible APIs using qryn.dev or qryn.cloud

But there are no straight up syslog/journald adapters?

qryn and vector get along very well! We use it all the time for testing and developing qryn and qryn.cloud and most of our users love it! But since qryn supports Logs, Metrics and Telemetry at once, we're just as compatible with any Agent supporting Loki/LogQL, Influx protocol for metrics and logs, Elastic Bulk, Prometheus for metrics, opentelemetry for everything... and more coming!

Feel free to open an issue on our repository if you end up trying it and/or would like us to help out!

https://qryn.dev https://qryn.cloud

Thank you. In this case,it was more a reference to:

> Our central syslog server is a central source of accessible, text based truth. It requires almost no infrastructure to be working on any machine in order to get logs to it and accept logs (...)

From the linked post on central log server (and Loki):

https://utcc.utoronto.ca/~cks/space/blog/sysadmin/GrafanaLok...

Unfortunately, AFAIK journald and qryn don't speak rsyslog - so you'll need something "alien" - all though vector seems very straightforward.

A couple years ago we were hurting from having no easily accessible centralized logs. We also had some in-house timeseries database tech that we were looking to deploy more widely but needed first to stress test.

After stumbling on an overview of Loki, we whipped up a service that presented a Loki-compatible API on top of our timeseries database, and killed two birds with one stone. Because we have full control over the data ingestion and storage too we are able to keep the data in a structured form with full type information.

It's fantastic in that it ties in with the rest of our use of Grafana too.

Loki is great but PromQL is a doozy. I’m basically useless on grafana without the query builder. Tempo is also pretty cool if you’re emitting otel spans.

I'm not that familiar with Loki but I'm aware that PromQL is the Prometheus Query Language. Just curious if PromQL works with Loki? Or are you referring to Prometheus being the more powerful system due to PromQL queries?

The Loki query language is heavily based on PromQL

Loki is the bomb-dizzle when it comes to real-time monitoring. First it’s designed to provide real-time monitoring of log data, allowing users to detect issues and troubleshoot problems as they occur. Gonna come in handy, particularly for mission-critical applications where downtime can have severe consequences.

Single node kube cluster running Loki and promtail and I’ve had great results until I started trying to push in some bare metal machine logs which cause 503 from nginx ingress. Then Loki stops working in Grafana.

But inside the cluster it’s great. Though my Loki helm chart is still on v3 rather than 4

We are hosting the grafana stack at my work. And if there are some comprehensive articles about trying to keep metrics counts low, and hosting on block storage with s3. I would appreciate peoples anecdata and links.

i am looking into loki to use at work. Just wondering do most of you self-host or use the grafana cloud?

Reply on news.ycombinator.com