This is a writeup I put together documenting how we (Snowpack Data) cut ETL costs by over 99% at Grid, and achieved an overall 80% reduction in BigQuery spend. Our blog documents the approach to the problems that arose in the ETL and provides an approachable deep dive into our implementation. The goal is to discuss the technical challenges without diving too deep into code examples, although we'd love to provide more technical commentary if anyone wants it!
Earlier this year I began a small boutique consulting company with the goal of leveraging my broad analytical experience with two dynamic partners to tackle challenges that growing businesses inevitably see in the analytics stack. The three of us had previously spent time at other startups, and most recently at Flexport, where we had realized the challenges that occur when short-term thinking in analytical foundations meet the realities of rapidly scaling businesses. We'll aim to continue to share our learnings and collaborate with others in the space as we continue to tackle these types of problems!