Thanks a lot for a detailed post. 20gb of log to not seem like much though? Not really sure of 500px scale. Fully conceding that at my company probably are logging too much, I'm wondering what takes ETL pipeline 4hr to run over that. Is just no need to optimize it since there's no benefit to have these metrics real time? Or am I missing some really data heavy part. Again not trying to be negative just wondering.
Luigi seems cool, can anyone comment on it compared Airflow or Spring XD, or is are those just different products.
Periscope looks like Kibana for relational stores, also cool to see.
Thanks again for great post!