HNHacker News
TopNewBestAskShowJobs

georgewfraser

2,609 karma · joined October 28, 2012

fivetran.com

twitter.com/frasergeorgew

george at fivetran dot com

submissionscomments
georgewfraser··on Clickhouse Local
All of the optimizations you describe are in every columnar analytical database built in the last 10 years, from Vertica to Snowflake.
georgewfraser··on Clickhouse Local
Snowflake, Redshift, BigQuery, Databricks, Presto.
georgewfraser··on Clickhouse Local
Why do people on HN love Clickhouse so much? As far as I can tell, it’s an ordinary column store, with a bunch of limitations around distributed joins and a heuristic-based query planner. There are several good analytical databases that will give you the same scan performance and a much better query planner and executor.

This is not a rhetorical question, I would really like to know why it gets so much attention here.

georgewfraser··on Modern Data Lakes Overview
Modern data warehouses (Snowflake, BigQuery, and maybe Redshift RA3) have incorporated all the key features of data lakes:

- The cost of storage is the same as S3.

- Storage and compute can be scaled independently.

- You can store multiple levels of curation in the same system: a normalized schema that reflects the source, alongside a dimensional schema that has been thoroughly ETL’d.

- Compute can be scaled horizontally to basically any level of parallelism you desire.

Given these facts, it is unclear what rationale still exists for data lakes. The only remaining major advantage of a data lake is that you aren’t subject to as much vendor lock-in.

georgewfraser··on Alphabet earnings show Google Cloud on $10B run rate
I would really love to know what the quarterly revenue is for just compute and storage, for each of the major cloud providers. Including things like G Suite and Office 365 just muddies the waters.
georgewfraser··on Doing a database join with CSV files
If your goal is to join large CSV files using a local program, the ideal tool is not SQLite but DuckDB:

https://www.duckdb.org/docs/current/sql/copy.html

The reason why DuckDB is a better fit for this job is because DuckDB is a column store and has a block-oriented vectorized execution engine. This approach is orders of magnitude faster when you’re doing batch operations with millions of rows at a time.

In contrast, SQLite would be orders of magnitude faster that DuckDB when you’re operating on one row at a time.

georgewfraser··on Tree-sitter: new incremental parsing system for programming tools (2018) [video]
It colors more accurately.
georgewfraser··on Tree-sitter: new incremental parsing system for programming tools (2018) [video]
The most obvious application of tree-sitter is editors. I wrote a VSCode extension to replace the built-in syntax coloring with tree-sitter-based coloring: https://marketplace.visualstudio.com/items?itemName=georgewf...

I actually think it would make more sense for the various VSCode language extensions to just bake in tree-sitter for their language. I have had a PR open to do this with golang for a while: https://github.com/microsoft/vscode-go/pull/2555

georgewfraser··on React Table is a “headless” UI library
I spent a ridiculous amount of time working on optimized table rendering in the early years of my company, and I came to a simple conclusion: if you want to display full-screen tables, with dense data, and scroll around fast, the only option is canvas. "Virtualized" HTML tables are always an order-of-magnitude slower, no matter how much effort you put into optimization.

The fundamental reason why you end up back at canvas is that if you are scrolling a full-screen table fast, even at 30 FPS you are going to have to rerender about half the table on each frame. So "virtualized DOM" doesn't really work; it's all about render speed. And canvas allows you to achieve unbeatable render speeds by specializing your drawing algorithm for your particular scenario.

georgewfraser··on ksqlDB: Event streaming database for stream processing applications
Streams are a fascinating topic, but this is not in any way shape or form a substitute for a real database. Take a look at the docs for joins:

https://docs.confluent.io/current/ksql/docs/developer-guide/...

It’s a long list of caveats about how your queries will silently fail if you don’t guarantee all sorts of properties of the data.

Anyone considering making Kafka the center of your data infrastructure should also consider using a conventional column-store database instead. The only advantage of Kafka in this comparison is better latency (seconds vs minutes). If you can live with a minute or two of latency, the advantages of a real database are MANY.

georgewfraser··on The Next 50 Years of Databases (2015)
You’re not going to run your analytical queries off your production DB once you have more than a couple users writing ad-hoc queries. And you probably don’t won’t to replicate your salesforce/ads/clicks into your prod db. So you’re going to create a replica. So why does it matter if the replica is a different type of database than prod?
georgewfraser··on How We Built a Vectorized SQL Engine
This is such a great post! Soooo many databases have implemented some kind of “column store extension” where they change the on disk representation and declare “look now we’re a hybrid database”. But the academic literature suggests that the bigger win for column stores may be in the execution layer. I love that Cockroach has taken the opposite of the usual approach and been so rigorous in measuring everything.

After doing all this work, is it your opinion that (hypothetically) a columnar on-disk representation would be a bigger or smaller win than a fully-built-our vectorized execution engine?

georgewfraser··on Building columnar compression in a row-oriented database
Thank you for responding. The “shallow and wide” query you describe is not a challenge for any recently built column store, which will partition the data into blocks of 10-100 MB. So when you run a query like:

select * from time_series where ingest_time > ?

You will only scan the most recent block, and it will be extremely fast. I have no doubt that there are individual queries where a time-series database will “win”, but I earnestly question whether there’s a real-world use case, with a mix of queries, where a special purpose time-series db is preferable to a good column store with time-based partitioning.

georgewfraser··on Building columnar compression in a row-oriented database
Reading this post is frustrating. What they are describing is where column store databases were 20 years ago. Perhaps at some point the folks at TimescaleDB will read Daniel Abadi’s 2008 paper, which describes the key elements of how all modern column stores work:

http://db.csail.mit.edu/pubs/abadi-column-stores.pdf

The key takeaway is that columnar compression only accounts for a small minority of the speed up that you get for scan-oriented workloads; the real big win comes when you implement a block-oriented query processor and pipelined execution. Of course you can’t do this by building inside the Postgres codebase, which is why every good column store is built more or less from scratch.

Anyone considering a “time series database” should first set up a modern commercial column store, partition their tables on the time column, and time their workload. For any scan-oriented workload, it will crush a row store like Timescale.

georgewfraser··on Waltz: A Distributed Write-Ahead Log
This design seems to be an example of a deterministic database system. There's an excellent review of deterministic databases here: http://www.cs.umd.edu/~abadi/papers/abadi-cacm2018.pdf

The core concept of all deterministic databases is simple: if your database is deterministic, multiple geographically-distributed replicas can execute the same transaction log independently, and they will all reach the same state. The problem of implementing a distributed database is reduced to implementing a distributed log.

However, there is one oddity of the Waltz design: the central locking mechanism. Other deterministic databases don't have anything like this, because it's not necessary. You can keep track of locks locally on each replica, and you can rely on every replica to reach the same conclusions about which transactions succeeded and failed, because they are deterministic.

Can anyone clarify why they are managing centralized locks?

georgewfraser··on Republican engineer fired by Google hits back
Why does google continue to have these internal message boards about non-work topics? What I’ve heard from friends at google is that discussion is dominated by a tiny, vocal minority with extreme views, much like a college campus. It seems like 99% of googlers and certainly the company would be better off if they just dropped the “bring your whole self to work” message and closed these forums.
georgewfraser··on Container Platform Security at Cruise
It seems weird to me that a self-driving car company is doing work on core server infrastructure things like container security. Does the builtin GCP stuff really not work for them? My impression has been that it’s really well thought-out.
georgewfraser··on A "cure" for baldness could be around the corner
Propecia is basically a cure for baldness as long as you start taking it before you lose too much hair. It’s very widely used—walk into any gay bar in San Francisco and you will notice there is hardly anyone under 40 who is bald.

There is a strange phenomenon online where in every discussion of propecia, someone will show up and claim they had terrible sexual side effects. But in the original clinical trials, the rate of such side effects was nearly the same as placebo. And I personally know about 10 men who take it and none have experienced side effects. I think people make these claims to get attention in forums, or it’s simply a placebo effect.

georgewfraser··on Streaming Cassandra at WePay
I guess I’m hoping a future version hides all the details of files and whatnot behind a high-level API. Something like:

select * from changes(tablex)

georgewfraser··on Streaming Cassandra at WePay
What does the 4.0 api look like? I would home that they would expose a high-level interface for change data, similar to SQL Server or Snowflake streams. Most databases leave this as an afterthought, which is so unfortunate.
georgewfraser··on Streaming Cassandra at WePay
There's some excellent choices in this design, but using Kafka in the middle makes no sense. The Cassandra change-data segments are written in batches, and BigQuery loads in batches. So the author is splitting up change-data segments into individual rows, feeding those rows into Kafka, and then re-batching them on the BQ side for ingestion.

The use of a streaming system adds a tremendous amount of complexity for no benefit, because the original data source is batch. A simpler solution would be to process the data in 5-minute batches and use a BigQuery MERGE statement to apply that "patch" to the existing table.

You really shouldn't use Kafka unless you have a true streaming data source and a hard latency requirement of < 5 minutes. It's much easier to work with data in small batches, using local disk and blob stores like S3 as your queue.

georgewfraser··on Vulnerability in the Mac Zoom client allows malicious websites to enable camera
Google meet uses UDP: https://support.google.com/a/answer/1279090?hl=en
georgewfraser··on Vulnerability in the Mac Zoom client allows malicious websites to enable camera
Why isn't zoom running fully in the web browser at this point? Meet does this, and as far as I can tell the quality is indistinguishable from Zoom. Can someone with a better understanding of the underlying protocols shed light on why Zoom continues to ship a separate desktop app?
georgewfraser··on Free Wolfram Engine for Developers
Wolfram language has a very different syntax that Matlab or really any other programming language, and it has a lot of lisp-like metaprogramming capability. It's not as well thought-out as a proper lisp like Clojure, it's verbose and there's a lot of weird corner cases.

The editor is very different as well: Mathematica pioneered the notebook concept that was copied by Jupyter, and the Mathematica user experience is still much more comprehensive and tightly integrated than anything else.

For linear algebra, they both use Intel MKL under the hood and get exactly the same performance.

georgewfraser··on Spina bifida: Keyhole surgery repairs baby spine in womb
The scenario I was describing is a little different: suppose we could detect spina bifida in an embryo before implantation. We could simply opt not to implant the embryo; it’s not the same as abortion, but it does prevent a person with spina bifida from existing. Is that bad in your way of thinking?
georgewfraser··on Spina bifida: Keyhole surgery repairs baby spine in womb
Your story is amazing thanks for sharing. Hypothetically, if spina bifida could be detected by pre-implantation genetic diagnosis, meaning that an embryo that was destined to be a person with spina bifida could simply not be implanted, would you also oppose pre implantation genetic diagnosis?
georgewfraser··on A Bunch of Money on AWS and Some Benchmark Results
There's a ton of gray area in benchmarking, you have to make a lot of little choices along the way, so I appreciate you sharing your reasoning.

> How come you haven't go to 10T?

We don't see a lot of 10 TB single datasets "in the wild". We think 100 GB - 1 TB is more representative of the average real-world data warehouse user.

> for us the warmup time separates the time to compile and code gen

You should include compile time---your compile time is really good! Redshift sometimes takes longer to compile the query than to run it, though I think they may have improved this in the last 6 months.

georgewfraser··on A Bunch of Money on AWS and Some Benchmark Results
As someone who's spent a lot of time working with TPC-DS [1] and talking to people about it [2], I see a couple areas that could be improved in this benchmark:

1. Total run time is not an appropriate way to summarize the performance across queries, because some queries take 100x longer than others. The appropriate way to summarize this kind of data is to use the geomean [3].

2. The official TPC-DS queries make heavy use of grouping sets, which are a rarely-used SQL feature. I think TPC-DS is better if you rewrite the queries to eliminate grouping sets.

3. You used the exact same queries to "warm up" the data warehouse, and to test the performance. Some data warehouses (notably Redshift) aggressively cache intermediate compilation results, so they are much faster the second time they see a query or even a fragment of a query. To model a real user submitting queries interactively, you should use warmup queries that are similar to but not the same as the ones you use to measure performance.

4. You can solve the "vendor benchmarking their own product" problem by submitting a PR to our repo [4], which currently tests Redshift, Snowflake, BigQuery, Azure SQL DW, and Presto. We'd be happy to review it and endorse the timing if it meets our standards for fairness!

[1] https://fivetran.com/blog/warehouse-benchmark

[2] https://www.youtube.com/watch?v=XpaN-PqSczM

[3] https://en.wikipedia.org/wiki/Geometric_mean

[4] https://github.com/fivetran/benchmark

georgewfraser··on Sisense Acquires Periscope Data
The last time I studied it closely was over a year ago, but at that point:

1. Periscope emphasizes directly writing SQL.

2. Looker hides the SQL behind "developer mode" and emphasizes clicking dimensions and measures.

3. Periscope supports R and Python somehow, though I haven't used it.

georgewfraser··on Pluralsight will acquire GitPrime for $170M
I stand by 100x in this context. OSS is way behind the commercial analytical databases. And deploying a horizontally scaleable analytical database is a completely different ballgame than, say, Postgres.
← PreviousPage 5 of 11Next →