Hacker News Reader

Comment by yevpats | Hacker News Reader

It's an interesting take but I worry this might not be possible (for a lot of cases) because not everything is database backed and sometime you have logic behind the API and you actually want the result of that API in your data warehouses. Think AWS API for example, even their own AWS Config team is using the same AWS APIs because there is no one place where the data just resides (of-course it would be great if it was the case, it would make life very easy)

I think this is a good tweet explaining the problem that will never be solved - https://twitter.com/mattrickard/status/1542193426979909634

Full Disclaimer: Im the founder and original author of https://github.com/cloudquery/cloudquery

View on HN

Very much hear you about the amount of logic contained within API layers. That said, it actually hasn’t come up a lot so far. What we’re finding is that a lot of teams that have complex API-layer logic also end up landing their data in a data warehouse and replicating the API’s data model there. In those cases, we can use that as the source. For those that don’t have a dwh, we do actually support connecting to an API as a source, it just requires a bit more config and it’s less efficient and so we don’t promote it quite as much :).

As far as the tweet goes, there’s a couple points we might disagree with. One of the assumptions made is that the problem is solved by an outsourced third-party. That’s actually exactly what we want to change! We think the problem should be solved by a first-party (the software vendor), and we want to give them the tools to accomplish that. Another assumption in there is that companies who charge for dashboards wouldn’t do this because they’d lose money. If they charge for dashboards, they can also choose to charge for data exports.

The API-based part sounds a lot link https://www.singer.io, the API-based data ETL tooling Stitch Data developed -- is that accurate?

There’s definitely some similarities (both are generic-ish ways to get data out of an API). As mentioned in another comment, we’re still deciding whether to adopt an existing protocol or roll our own for those API connections. We’ve done the latter so far but it’s a work in progress.

I very much hope it will work for you and it will succeed. This can be def an amazing turning point in data integration and ELT world.

I do still have hard time understanding how this is technically feasible because let's say a company have their data stored in PostgreSQL but the data model their is not exactly as in their API because they are doing some minimal transformation and then expose it to the user. How the user will know what to expect the new data model is not documented at all and they usually want what is documented and exposed via the API. so seems you still have to go via the API and if you are there already then your are in the ELT space.

Per your comment that it falls into the user and it should fall on the vendor - I agree but if it's a matter in the ELT space then they should just maintain a plugin to CloudQuery (https://github.com/cloudquery/cloudquery) or AirByte or something similar. Similarly how vendors maintain terraform or pulumi plugins.

Reply on news.ycombinator.com