HNHacker News
TopNewBestAskShowJobs

georgewfraser

2,609 karma · joined October 28, 2012

fivetran.com

twitter.com/frasergeorgew

george at fivetran dot com

submissionscomments
georgewfraser··on Citus 10 brings columnar compression to Postgres
Beware that simply adding a column-oriented storage engine to a row store like Postgres is not going to get you anywhere near the performance of a ground-up columnar system like Redshift or Snowflake. This paper explains why [1]. Short version: most of the benefits are in the column-oriented execution engine, which differs in every aspect of its implementation from a row-oriented execution engine.

[1] https://stratos.seas.harvard.edu/files/stratos/files/columns...

georgewfraser··on Citus 10 brings columnar compression to Postgres
Do you have a source for this, or a code sample that can demonstrate it? This would be an extremely naive implementation of columnar storage. There are some truly hard cases around long variable-length strings, but any halfway decent columnar storage engine should be able to handle columns with different widths.
georgewfraser··on A Data Pipeline Is a Materialized View
Great article, one quibble: there isn’t really a clear dividing line between batch and streaming. If you process data one row at a time, that is clearly a streaming pipeline, but most systems that call themselves streaming actually process data in small batches. From a user perspective, it’s an implementational detail, the only thing you care about is the latency target.

Nearly all data sources, including the changelogs of databases, are polling-based APIs, so you’re getting data from the source in (small) batches. If your goal is to put this data into a data warehouse like Snowflake, or a system like Materialize, the lowest latency thing you can do is just immediately put that data into the destination. I sometimes see people put a message broker like Kafka in the middle of this process, thinking it’s going to imbue the system with some quality of streamyness, but this can only add latency. People are often surprised that we don’t use a message broker at Fivetran, but when you stop and think about it there’s just no benefit in this context.

georgewfraser··on Apache Arrow 3.0
Much of the value of Arrow is in the things that will get built after Arrow is widely supported by data warehouses. Much of the data ecosystem we have today was designed to avoid the cost of moving data between systems. The whole Hadoop ecosystem is written in Java and shoehorned into map-reduce for this reason.

Imagine if, for example, you could use Mathematica or R to analyze data in your Snowflake cluster, with no bottleneck reading data from the warehouse even for giant datasets. This is the future that’s going to be enabled by Arrow.

georgewfraser··on Apache Arrow 3.0
The main benefit of a columnar representation in memory is it's more cache friendly for a typical analytical workload. For example, if I have a dataframe:

  (A int, B int, C int, D int)
And I write:

  A + B
In a columnar representation, all the As are next to each other, and all the Bs are next to each other, so the process of (A and B in memory) => (A and B in CPU registers) => (addition) => (A + B result back to memory) will be a lot more efficient.

In a row-oriented representation like protobuf, all your C and D values are going to get dragged into the CPU registers alongside the A and B values that you actually want.

Column-oriented representation is also more friendly to SIMD CPU instructions. You can still use SIMD with a row-oriented representation, but you have to use gather-scatter operations which makes the whole thing less efficient.

georgewfraser··on Apache Arrow 3.0
Arrow is the most important thing happening in the data ecosystem right now. It's going to allow you to run your choice of execution engine, on top of your choice of data store, as though they are designed to work together. It will mostly be invisible to users, the key thing that needs to happen is that all the producers and consumers of batch data need to adopt Arrow as the common interchange format.

BigQuery recently implemented the storage API, which allows you to read BQ tables, in parallel, in Arrow format: https://cloud.google.com/bigquery/docs/reference/storage

Snowflake has adopted Arrow as the in-memory format for their JDBC driver, though to my knowledge there is still no way to access data in parallel from Snowflake, other than to export to S3.

As Arrow spreads across the ecosystem, users are going to start discovering that they can store data in one system and query it in another, at full speed, and it's going to be amazing.

georgewfraser··on Databricks is an RDBMS
Because it actually costs the same, and if you process them in Snowflake using SQL or UDFs, you will get your results in seconds and you won't have to manage any of the underlying infrastructure.
georgewfraser··on Databricks is an RDBMS
EXACTLY. You absolutely can store unstructured and semi structured data in Snowflake. I find it baffling and at this point a bit irritating that there is this community of people insisting that is not allowed for...some unspecified reason.
georgewfraser··on Databricks is an RDBMS
If Teradata is faster for workload X or Y, it isn’t because of a shared nothing architecture. Databricks and Snowflake both cache the working set on local disks of the worker nodes, so in practice they are shared nothing systems from a performance perspective.
georgewfraser··on Databricks is an RDBMS
Interestingly, BigQuery is also missing multi-table transactions. There are ways to live without this feature, but I agree it's a gap.
georgewfraser··on Databricks is an RDBMS
It's a lot simpler to use a single system as both your data lake, and your data warehouse. As Databricks gets better and better at the core data warehouse features, it becomes feasible to use it for both. Meanwhile, Snowflake and BQ are coming from the other direction, implementing data lake features. AWS strategy seems to be, just make it easier to have 2 systems and move data back and forth.
georgewfraser··on Launch HN: Airbyte (YC W20) – Open-Source ELT (Fivetran/Stitch Alternative)
The fundamental challenge of open-source ETL is that high-quality connectors require understanding and working around all kinds of corner cases in the API of each data source. It’s very hard to get open source contributors to do this kind of work; it’s a real slog. Hence at Fivetran we’ve always stuck with the commercial route.
georgewfraser··on Waymo CEO dismisses Tesla self-driving plan: “This is not how it works”
My point is that it doesn’t fail in edge cases. It has very clear limitations, there are no surprises. Even if you were literally asleep at the wheel, the worst thing that would happen is it’d end up stopped in front of a traffic light, waiting for you to take over.
georgewfraser··on Waymo CEO dismisses Tesla self-driving plan: “This is not how it works”
I have a model Y and what it has today is 80% as useful as a completely self driving car. It’s mostly self driving on the highway, and the situations it can’t handle are very predictable: toll booths, stopped cars in the middle of the road, things like that. You need to available to drive but you always have plenty of warning before you need to take over.
georgewfraser··on Loon’s final flight
This kinda tells you everything about the culture of Google:

> Sadly, despite the team’s groundbreaking technical achievements over the last 9 years — ... — the road to commercial viability has proven much longer and riskier than hoped.

Focus is on the technical achievements, never mind if anyone ever used it.

georgewfraser··on Ballista: Distributed compute platform implemented in Rust using Apache Arrow
These new GCs are amazing technology, but they primarily target pause time, whereas in data processing the primary concern is the “headroom” of extra space in your heap to allow the GC to work efficiently.
georgewfraser··on Ballista: Distributed compute platform implemented in Rust using Apache Arrow
This kind of data infrastructure is a great use case for Rust. A lot of data infrastructure is memory-bound, so saving the memory overhead of GC is a huge win.

The use of Arrow to support multiple programming languages is also a great concept. Other distributed computing engines have ended up tied to the JVM (Spark, Presto, Kafka) as a way of avoiding serialization/deserialization costs when you go across a language boundary. Arrow is a really elegant solution, as long as you're willing to batch up operations.

georgewfraser··on Lakehouse: New Open Platforms That Unify Data Warehousing and Advanced Analytics [pdf]
What is not said in this article is that you can use modern data warehouses, like Snowflake and BigQuery, in the exact same way: a single system that serves as both your data lake and your data warehouse. Databricks and the cloud data warehouses are rapidly converging. Databricks has enough SQL functionality that it can be reasonably be called an RDBMS, and Snowflake has demonstrated that you can incorporate the benefits of a data lake into a data warehouse by separating compute from storage. At this point, the main difference is that with Databricks you can directly access the underlying Parquet files in S3. Does that matter? For some users, yes.
georgewfraser··on MicroStrategy's Bitcoin Debt Bet
This is really bizarre. MicroStrategy is a solid BI tool, it's been solid for a long time, and the company basically operates as a cash machine. This is when you start issuing a dividend. If the founder-CEO is bored, he should promote one of the many people who have made their careers at MicroStrategy and retire.

Instead, he's decided to use the existing business he founded decades ago as an outlet for his enthusiasm for Bitcoin. This is why God created activist investors, I think.

georgewfraser··on Kafka Is Not a Database
"Read-committed isolation" is not a meaningful implementation of transactions. If you can't do read, then a write, while guaranteeing the database didn't change in between, then you don't really have transactions.
georgewfraser··on Kafka Is Not a Database
We're trying to address a real problem that is happening in our industry: VPs of eng and principal engineers at startups are adopting the "Kappa Architecture" / "Turning the Database Inside Out", without realizing how much functionality from traditional database systems they are leaving behind. This has led to a barrage of consistency bugs in everything from food-delivery apps to the "unread message count" in LinkedIn. We're at the peak of the hype cycle for Kafka, and it's being used in all kinds of places it doesn't belong. For 99% of companies, a traditional DBMS is the right foundation.
georgewfraser··on Kafka Is Not a Database
That post explains that there are scenarios where it makes sense to store data permanently in Kafka. "Kafka is Not a Database" makes a different point, which is that Kafka doesn't solve any of the hard transaction-processing problems that database systems do, so it's not an alternative to a traditional DBMS. This is not a straw man---Confluent advocates for "turning the database inside out" all over their marketing materials and conference talks.
georgewfraser··on Kafka Is Not a Database
It's funny that you use that example, we actually cited that in an earlier draft of this post. Despite the seemingly opposite title "Queues are Databases", that note actually makes many of the same arguments, that message brokers are missing much of the functionality of database management systems and this is a problem.
georgewfraser··on Materialize Raises a $32M Series B
Joins were unavailable or subject to extreme limitations. Or just plain wrong!
georgewfraser··on Materialize Raises a $32M Series B
Materialize has tackled the hardest problem in data warehousing, materialized views, which has never really been solved, and built a solution on a completely new architecture. This solution is useful by itself, but I'm also watching eagerly how their road map [1] plays out, as they go back and build out features like persistence and start to look more like a full-fledged data warehouse, but one with the first correct implementation of materialized views.

[1] https://materialize.com/blog-roadmap/

georgewfraser··on The Covid Science Wars
The thing that has really surprised me is that public health officials are expressing so much confidence that masks work, despite the evidence being quite meh [1]. There have been many attempts to demonstrate that masks prevent transmission of flu and overall the results are weak/negative. The focus on masks has trade offs. People have finite attention, so the focus on masks diminishes the attention we apply to other interventions.

[1] https://www.publichealthontario.ca/-/media/documents/ncov/co...

georgewfraser··on AWS releases Glue Databrew, a visual ETL tool
Visual ETL is not as good of an idea as it seems at first. You end up putting a ton of business logic into the menus of these tools, and it’s not version controlled, and it’s not searchable. You’re better off doing this kind of work in SQL.
georgewfraser··on NoisePage – Self-Driving Database Management System
Welcome to HN Andy! As you are witnessing, people find that actually reading the links can be a real impediment to hot takes...
georgewfraser··on Dark’s new backend will be in F#
There’s F# language server, which is not as featureful as ionide but a lot faster.
georgewfraser··on My Resignation from the Intercept
The DKIM headers from the emails have been verified, so they are real: https://twitter.com/erratarob/status/1322007153415200768?s=2...
← PreviousPage 3 of 11Next →