Ask HN: Log Analytics or Data Warehouse?
It was suggested that I consider a log-oriented system instead like ELK or Splunk/Graylog/Sumo Logic.
There are many dimensions we're looking at like query language and how we can staff around that choice, data volume, BI tools, alerting, costs for setup and management, etc.
Our data includes some system logs but we're mostly serverless, App logs, ALB access logs, some batch flat files, and streams of webhooks from various vendors.
We want to be as close to NoOps as we can afford to support the system.
I've found a remarkable amount of cross-over between log analysis tools and data warehouses. I see that log tools tend to be much faster, but lack native joins, requiring ETL work, as well as scale/data recency decision to make. BigQuery specifically seems to alleviate a lot of operational challenges and can do ETL internally once Stackdriver streams into it. I've ran with both and found both can meet my needs with a few niggling exceptions.
If you have had to make a similar decision I would really appreciate your feedback.