We used Grafana + KairosDB and turned the logs into tags essentially.
Our entire productions system had easy 'taps' that you could annotate to monitor everything about our crawler.
For example, number of HTTP requests, their status codes, the language of the content.
We also record intersections of the tags like lang+domain.
The downside of a system like this is that you have to know all your metrics apriori... If not and you need them at runtime you're out of luck.
The UPSIDE is that you use like 1/100th of the total size you would originally need for raw logs.
What we found is that you quickly converge on the tags you need and then you don't end up adding many more.
Everyone says 'disk' is cheap but in our situation our logs outpace the amount of data we would collect. We'd have 1000s of petabytes of logs by now.