A modern data stack for startups (2022)
incident.io
incident.io
Worth some observations that:
- We're still using Fivetran for the EL stages. Costs are much more significant than they were before and we're looking (for the high volume sources) into options like DataStream as cost savers, but it's not unmanageable.
- dbt is still working great, even if we've done a lot of investment having now built a 5 person data team (BI, DA, DE) around it.
- Still use Metabase but have some frustrations and are considering other options.
- We no longer use Stitch :tada:
There's a post that followed this on improvements we made to our setup that may be interesting: https://incident.io/blog/updated-data-stack
The OP is still full of relevant, useful information, though (imo, of course).
Note that we build dashboards for customers that end up having some complex SQL directly in Superset.
The data that goes into those insights can be quite complex and the queries are actually threading JSON parameters through into BigQuery SQL queries using JavaScript UDFs to power the filters in the dashboard (show incidents with these custom field values). This works pretty well with signed Metabase dashboard links.
We have hit limitations with Metabase though. Performance of the instance can be a bit unpredictable and their support has been poor when things do go wrong, with very little willingness to take our feedback into account for new product features.
For that reason and more (such as more flexible dashboards) we’re going to move ourselves to Omni (https://omni.co/) for internal business analytics use cases, and will reconsider Metabase for our customer facing product dashboards when we do. Omni may work for these or we might build them bespoke, we’ll see at the time.
For anyone who doesn't know: https://superset.apache.org/
(There's at least one service that offers managed Superset hosting if that's what you're looking for; it's easy to find so I won't link it here.)
It looks like a promising tool though! I’m sure we’ll blog about our experience with the new tooling once we’ve moved over, the team will no doubt have a lot to say about it.
From their website[0]:
> Preset was founded by the original creator of Apache Superset™. Our team of experts contributes over 75% of all commits to the open-source software project.
I'd be interested to see your blog post, regardless of tool.
Superset the best of them at data visualization but I honestly found it almost useless for self-serve BI by business users if you have existing star schema. This issue on how to do joins in Superset (with stalebot making a mess XD) is everything difficult about Superset for BI in a nutshell. https://github.com/apache/superset/issues/8645
Metabase is pretty great and it's definitely the right choice for a startup looking to get low cost BI set up. It still has a very table centric view, but feels built for _BI_ rather than visualization alone.
Lightdash has significant warts (YAML, pivoting being done in the frontend, no symmetric aggregates) but the Looker inspiration is obvious and it makes it easy to present _groups of tables_ to business users ready to rock. I liked Looker before Google acquired it. My business users are comfortable with star and snowflake schemas (not that they know those words) and it was easy to drop Lightdash on top of our existing data warehouse.
I've not worked at any startups that did data warehousing, the one place I did work at where we were /starting/ to get it setup was like 300+ employees and $100M+/year revenue.
We also use the same data pipeline to power a lot of our data product features which customers pay us for.
So it’s extremely worthwhile as an investment for us. It’s also why we have about five people hired into data adjacent roles, as it’s so key to us running the business correctly.
We also have some parts of our dataset that are complicated to compute.
An example is that we estimate the hours that people have spent working on incidents, with a method that we describe here: https://incident.io/guide/insights/workload
That requires us to do some expensive cross-joining of every action ever taken in an incident from messages sent to the channel to GitHub PRs being merged. We could make this incremental and optimise it for performance but using BigQuery by default means we don't need to worry yet and can leave the optimisations for when we're bigger and the engineering resource we'd dedicate wouldn't detract as much from customer-focused work.
ETL should be minimized (except for external data, which is a bad sign of data owned or managed by a third-party) and replaced with the equivalent of dynamic or materialized "views". Prefer to create hygienic "views" of data against original data rather than mutating and destroying such original data with destructive transformations.
Finally, have a deeply-integrated, robust, enterprise-wide, fine-grained ACL system and privacy policy to keep everyone (and system users) from accessing anything without a specific business purpose need and an approval audit record stored via some sort of blockchain-like tech.
Obviously very different resource constraints than Meta, so worth considering which situation you may be closer to when picking an implementation plan.
Plan A: $16 (month/user)
Plan B: $10,000+ Call Us
Plan C: Call Us
Those are some of the steepest price cliffs I’ve ever come across.
The Pro plan contains a load of features like insights, audit logs, API access, etc that you don't get in Starter, hence the increase per-seat bill.
Would be interested if that makes more sense to you? (I work there, always keen to hear feedback)
30 responders probably covers a company of 100+ people (viewers are free after all). If you are smaller but want things like an API or webhooks, the minimum payment for the Pro plan is a huge cliff. And it doesn't even give you audit logs or an SLA.
Many Cyber Insurance policies mandate SSO
The SAML feature in the Pro and up plans is about hooking us up directly to your identity provider to do things like RBAC and group syncing, and we pay WorkOS a monthly fee of several hundred dollars per customer that uses this, so makes sense to gate in the higher plans.
So I’m not sure this is quite right?