In Kibana, if something is there I will find it with ease and it doesn't take a lot of time to investigate issues in a microservice based application. It is also quite fast.
So good but not perfect! When we have the time we'll look for alternatives
2. It has a convenient and simple query language.
3. It works very well with traces and metrics.
the pain part:
1. It struggles to query logs over a wide time range.
2. Its indexing (or labeling) capabilities are very limited, similar to Prometheus.
3. Due to 1 and 2, it is difficult to configure and use correctly to avoid errors related to usage limits (e.g., maximum series limits).
IMHO, Loki query language is the most inconvenient language for logs I've seen:
- It doesn't support calculating multiple stats in a single query. For example, it cannot calculate the number of logs and the average request duration in a single query.
- Its' syntax for aggregate functions is very unintuitive and is hard to use, especially if you aren't familiar with PromQL.
- It requires putting an annoying "|=" separator between words and phrases you are searching in logs.
- You need to use a hack with JSON parsing when filtering or stats calculations on log fields is needed.
Modern columnar SQL such as ClickHouse are 10+ times more efficient in real-world use cases.
I'm a CEO and founder of Quesma, which, let's use Kibana with ClickHouse: https://quesma.com/
Forever free, source-available license.
Also out of box configuration sinks 1TB/hr quite happily in microservices mode.
Could you share Loki config, which can deal with 1TB/hr volume of logs?
why do you need storing JSON-encoded string inside log field? It is much better parsing the JSON into separate fields at log shipper and storing the parsed log fields into Elasticsearch. This gives better query performance and may also reduce disk space usage, since values for every parsed field are stored separately (this usually improves compression ratio and reduces disk read IO during queries if column-oriented storage is used for per-field data).
I tried explaining this at https://itnext.io/why-victorialogs-is-a-better-alternative-t...
I was not able to do that with the log shipper. If I configured parsing, then the not-JSON messages would get dropped.
Elastic was kind of a resource hog and much more expensive for the same amount of data.
That might be dependent on your use case though.
For grafana cloud and Loki you can close to a good usability with LBAC (label based access control) but you still need have many data sources to map onto each “team view” to make it user friendly.
What is missing for me is like in elastic a single datasource for all logs which every team member across all teams can see and you scope out the visibility level with LBAC