I worked for a large organisation where management was far closer to 'technology leaders' and 'technology strategists' than engineering and data science principles and leads. They would endlessly swoop in to our division asking us to assess another product they have bought to fix the legacy problems of multiple data sources.
All of them were brittle af. They all anticipated a very idealistic data source and the absence of non-technical people curating data in excel ten different ways.
Even though we were the data science team, we usually ended up providing far more value to the organisation because we could do data engineering and cleaning and ended up being the source of truth for a lot of data required by the wider organisation. We got pitched dozens of sexy solutions to fix all our ETL problems, but when we started asking questions it was always seemed like a well designed custom pipeline couldn't be beaten for both data quality assurance, reliability and speed.
Instead Airbyte gets adopted by engineers, data scientist... to solve one problem and then the usage expands from there. We can improve the product based on the feedback we get from the real users.
And if a feature, a connector is not there, anyone can actually add it!
Warehousing that data might also require firewalling clients or teams for privacy or “competitive/conflict” reasons.
These aren’t difficult problems to solve with a few knowledgeable devs but that is nothing but added cost and some agencies just aren’t good at hiring the right devs - especially if their previous exposure has been basic front end web developers from their clients.
“Data warehouse” has also become a selling term even if “really big database” is a more accurate term.
Hopefully more of these companies start to distinguish themselves in this space but their competition isn’t each other - it’s entry-level data people blasting through Excel.
people hate hiring data engineering (plumbing people feels like cost), and data eng like tools that work but most are.too niche/happypath-oriented, so even w trifacta etc, a lot of open territory. SW can solve a lot of that, in theory, so everyone wins.
And I agree that until there is an oss winner, the proprietary stuff will keep getting churned through. So ultimately whatever your data platform does (aws, databrick, whatever) or oss you're bringing. A lot of room for vendors to carve out niches b/c of connectors x use cases, until platforms/oss eats them all. VC's will see some ARR and name brands and thus be happy to fund: a lot of gaps any startup can fill. (I am impressed by airbyte for a few non-technical reasons even without having used it, so not a knock on them, so just some clues for the continuing froth in their market.)
Maintaining connectors is also a good way to demotivate high achievers - better to have them further down the value funnel.