Hacker News Reader

Comment by WinonaRyder | Hacker News Reader

We love JSON logs and previously just sent most of it to systemd's journald and use a custom tool to view them. But maybe a year ago Grafana released https://github.com/grafana/loki and we've been using it on https://oya.to/ ever since.

IIRC, the recommended way to integrate it with Grafana is via promtail but we weren't too keen on the added complexity of yet-another service in the middle so we developed a custom client library in Go to just send the logs straight to Loki (which we should probably open source at some point).

I don't think there's any fancy graph integration yet, but the Grafana explore tab with log level/severity and label filtering works well enough esp. since they introduced support for pretty printed JSON log payloads.

View on HN

I have tried out loki, too. But I was not satisfied because you have to run an extra server for it. I have only a very small app so I was searching for a much simpler solution and found https://goaccess.io/ . The nice thing is that it is very flexible (you can pipe your logs in command line, but also run as a server) and if you are using standard tools like e.g. nginx or apache, the setup only takes an evening :)

Here are a few more monitoring tools listed https://turtle.community/topic/monitoring

It is always a question of complexity and how much time you want to spend :) At first I used Multitail to peek into server logs via ssh. Then I switched to GoAccess and if you really have a greater infrastructure I would maybe switch to Loki or ELK.

I thought goaccess (which is spectacular) was for HTTP logs, and not a general purpose logging solution

I've used GoAccess in the past, but didn't find it a good fit esp. since I prefer JSON logs. I don't know if it supports this - maybe I didn't read enough docs - but at the time, I remember it was easier to spend an hour whipping up a tool in Go that did exactly what I wanted.

As for loki, it's a separate server but the setup takes maybe 10-30 minutes of reading some docs, maybe changing some config files and the systemd unit file to keep it up and running is less than 10 lines (most of which is boilerplate).

Of-course I have the benefit of a client library so I can just call a function on a struct at the end of a request with no need to worry about serializing the relevant data into some predetermined format, compression, etc.

> I've used GoAccess in the past, but didn't find it a good fit esp. since I prefer JSON logs. I don't know if it supports this

Yes, GoAccess supports HTML, JSON and CSV: goaccess --output=json

> we developed a custom client library in Go to just send the logs straight to Loki (which we should probably open source at some point).

I'd be interested in seeing this, we're likely to start using Loki at Sourcegraph soon and would likely want this approach as well.

I'll push it to https://github.com/oyato/promqueen when it's ready. Hopefully I will have time to work on it over the next week or two.

Awesome! Much appreciated :)

Reply on news.ycombinator.com