To get the performance run more queriers (horizontally scaled read path that is in front of the object storage).
We're improving performance constantly, and we already run Loki at a very large scale (multi-region, multi-tenant, etc) with aggressive internal SLOs. We see customers doing network analytics, log analytics, line-of-business data analytics all on Loki and it works really well. Customers also use it as part of Cloud hosted ETL processes and to drive alerts. What we run in the Cloud today is the OSS version on k8s.
That said... if we're missing some use-case or just a scenario that you feel should work and it doesn't work for you, then we really want to learn about it. If there's an out of the box experience that isn't great, then let us know. Feel free to ping me any details you're not willing to put on HN at david.kitchen@grafana.com but I'm very cool with public conversations too.
I have a question. I have put some time trying to learn the Grafana stack. Heaviness of elastic both self hosted and cloud version has led me to seek for alternatives. Is there a way to get logs into Loki that can be run offline, when jobs have finished and hosts have been shutdown? It seems Loki recommends Promtail agent which looks a bit heavy-handed (https://grafana.com/docs/loki/latest/getting-started/get-log... )
I have relaxed requirements on gathering logs for analytics. I’d much rather just run a tool that goes through log files and ingests them into Loki. Results of several jobs would be compared as a whole.
From what I’ve seen Loki with Grafana can do a lot. For example this video: https://m.youtube.com/watch?v=7zmRhHd-ohk
Thank you!
```cat <files> | promtail -stdin```
Though we like the idea as a feature request... "Have promtail support reading a file or directory and exiting once complete".
So we've put that on the plan.
Agree that the name isn't ideal as it's not clear that this is a "tail and send text files" process.
But yup, Promtail is what you want.