ctc24

Appreciate that and will do!

Agreed! It's kind of fascinating that a better standard hasn't emerged yet, but not surprising given that each one is developed in isolation by a different organization.

Definitely a lot of room for improvement. Curious if anyone has thoughts about the best route to getting such a standard / protocol in place: it seems like a lot of stars would have to align, but would be invaluable nonetheless.

Understand the sentiment. On the flip side, and as noted in the post, we don't want it to serve as a "name and shame" situation. We have a lot of respect for the folks maintaining the drivers we use, they do a pretty hard job, and so we don't want to come out swinging and taking shots at them. So we're trying to walk that line.

We also think the information provided in the post is valuable as is: it's easy enough to check whether a driver you use faces any of the issues we mention.

With that said, here's a bit more color:

- the native Snowflake driver for GoLang does not implement COPY functionality (or at least it did not when we last tried to use it).

- the memory leaks are pretty prevalent across ODBC drivers. It's worth watching out for that one if you're using any ODBC driver.

- the breaking change on connection string was Databricks' GoLang driver.

- the DECIMAL one is pretty prevalent too. BigQuery only allows you to go up to DECIMAL(38,9) while most other drivers let you go to 18 on scale, and ClickHouse supports precision/scale of up to 76. Redshift complains loudly if you try to insert a DECIMAL(38,17) into a DECIMAL(38,18) column, for instance.

Hope the added color is helpful!

edit: formatting

Thanks! In the case of traditional ETL tools, a lot of that transformation is usually schema cleaning, which is necessary because the data is extracted from an API and so needs a little love before it’s in analysis-ready shape (renaming columns, joining a couple tables, and so on). When the vendor is in charge of defining the data model, they know what format is most helpful to surface the data in and so can transform it to that format before the load (TEL?!).

In this model, it’s also still possible for the data team on the recipient side to further transform data before handing it to value teams. They can run dbt or whichever transform tool they use once it lands in their warehouse (so more like ELT, which is broadly what the industry is moving to).

1. I think in practice, people are already using a mix of sources for ingestion today. It’s rare that a data team would rely on a single tool to ingest all their data – instead, they might get some data via one or more ETL tools, some data via a script they wrote themselves, and some other data from their own db. So in that regard, I don’t think a world where the vendor provides the data pipeline makes this a lot more complex.

One way we’re hoping to pre-empt some of this is by helping vendors to surface more observability primitives in the schema that they write data to. To give an example: Prequel writes a _transfer_status table in each destination with some metadata about the last time data was updated. The goal there is to decouple the means of moving data with the observability piece.

We can also help vendors expose hooks that people’s data pipelines & observability tools plug into (think webhooks and the like).

2. We don’t really – anecdotally, companies that offer data warehouse syncs tend to be pretty focused on providing a great user-experience. At the end of the day, that decision is pretty much entirely with the business. We see a pretty wide range today: some teams choose to make exports widely available, some choose to reserve it for their pro or enterprise tier, and some choose to sell it as a standalone SKU. It’s pretty similar to what already happens with APIs designed for data exports.

Would love to continue the convo and hear more about your take on obs! Sending you a note now.

As you can imagine, the underlying tech is pretty much the same – we’re still moving bits from one db/dwh to another – so we can use our existing transfer infra. The only difference is the UX and API we surface on top of it.

We got this request from a couple teams we work with, and so we have an alpha that’s live with them. It’ll probably go live in GA in Q4 or Q1 (and if you want access to it before then, drop us a line at hn (at) prequel.co!).

There’s definitely some similarities (both are generic-ish ways to get data out of an API). As mentioned in another comment, we’re still deciding whether to adopt an existing protocol or roll our own for those API connections. We’ve done the latter so far but it’s a work in progress.

That’s something that we spend a lot of time working on. Our general approach is to do both the simple and predictable thing (ie what format would I want to get the data in if I were the data team receiving it). For the specific example you give, we’d convert the timestamptz into a timestamp before writing it to the destination.

Another type where this comes up is JSON. Some warehouses support the type whereas others don’t. In those cases, we typically write the data to a text/varchar column instead.

The way it works under the hood is we’ve effectively written our own query planner. It takes in the source database flavor (eg Snowflake) and types, the destination database flavor, and figures out the happy path for handling all the specified types.

That’s actually pretty close to how we connect to APIs today, but I love the idea of streamlining it as a protocol. We’re still working on improving the dev experience, especially around APIs, and would love to chat more if you have more feedback!

Very much hear you about the amount of logic contained within API layers. That said, it actually hasn’t come up a lot so far. What we’re finding is that a lot of teams that have complex API-layer logic also end up landing their data in a data warehouse and replicating the API’s data model there. In those cases, we can use that as the source. For those that don’t have a dwh, we do actually support connecting to an API as a source, it just requires a bit more config and it’s less efficient and so we don’t promote it quite as much :).

As far as the tweet goes, there’s a couple points we might disagree with. One of the assumptions made is that the problem is solved by an outsourced third-party. That’s actually exactly what we want to change! We think the problem should be solved by a first-party (the software vendor), and we want to give them the tools to accomplish that. Another assumption in there is that companies who charge for dashboards wouldn’t do this because they’d lose money. If they charge for dashboards, they can also choose to charge for data exports.

Hey team, Charles here, co-founder of prequel.co (YC W21). We also help companies share data with their customers.

As someone who's been playing in the data sharing space, it's really exciting to see it get more attention!

All the best to y'all!