2,609 karma · joined October 28, 2012
twitter.com/frasergeorgew
george at fivetran dot com
This is not a rhetorical question, I would really like to know why it gets so much attention here.
- The cost of storage is the same as S3.
- Storage and compute can be scaled independently.
- You can store multiple levels of curation in the same system: a normalized schema that reflects the source, alongside a dimensional schema that has been thoroughly ETL’d.
- Compute can be scaled horizontally to basically any level of parallelism you desire.
Given these facts, it is unclear what rationale still exists for data lakes. The only remaining major advantage of a data lake is that you aren’t subject to as much vendor lock-in.
https://www.duckdb.org/docs/current/sql/copy.html
The reason why DuckDB is a better fit for this job is because DuckDB is a column store and has a block-oriented vectorized execution engine. This approach is orders of magnitude faster when you’re doing batch operations with millions of rows at a time.
In contrast, SQLite would be orders of magnitude faster that DuckDB when you’re operating on one row at a time.
I actually think it would make more sense for the various VSCode language extensions to just bake in tree-sitter for their language. I have had a PR open to do this with golang for a while: https://github.com/microsoft/vscode-go/pull/2555
The fundamental reason why you end up back at canvas is that if you are scrolling a full-screen table fast, even at 30 FPS you are going to have to rerender about half the table on each frame. So "virtualized DOM" doesn't really work; it's all about render speed. And canvas allows you to achieve unbeatable render speeds by specializing your drawing algorithm for your particular scenario.
https://docs.confluent.io/current/ksql/docs/developer-guide/...
It’s a long list of caveats about how your queries will silently fail if you don’t guarantee all sorts of properties of the data.
Anyone considering making Kafka the center of your data infrastructure should also consider using a conventional column-store database instead. The only advantage of Kafka in this comparison is better latency (seconds vs minutes). If you can live with a minute or two of latency, the advantages of a real database are MANY.
After doing all this work, is it your opinion that (hypothetically) a columnar on-disk representation would be a bigger or smaller win than a fully-built-our vectorized execution engine?
select * from time_series where ingest_time > ?
You will only scan the most recent block, and it will be extremely fast. I have no doubt that there are individual queries where a time-series database will “win”, but I earnestly question whether there’s a real-world use case, with a mix of queries, where a special purpose time-series db is preferable to a good column store with time-based partitioning.
http://db.csail.mit.edu/pubs/abadi-column-stores.pdf
The key takeaway is that columnar compression only accounts for a small minority of the speed up that you get for scan-oriented workloads; the real big win comes when you implement a block-oriented query processor and pipelined execution. Of course you can’t do this by building inside the Postgres codebase, which is why every good column store is built more or less from scratch.
Anyone considering a “time series database” should first set up a modern commercial column store, partition their tables on the time column, and time their workload. For any scan-oriented workload, it will crush a row store like Timescale.
The core concept of all deterministic databases is simple: if your database is deterministic, multiple geographically-distributed replicas can execute the same transaction log independently, and they will all reach the same state. The problem of implementing a distributed database is reduced to implementing a distributed log.
However, there is one oddity of the Waltz design: the central locking mechanism. Other deterministic databases don't have anything like this, because it's not necessary. You can keep track of locks locally on each replica, and you can rely on every replica to reach the same conclusions about which transactions succeeded and failed, because they are deterministic.
Can anyone clarify why they are managing centralized locks?
There is a strange phenomenon online where in every discussion of propecia, someone will show up and claim they had terrible sexual side effects. But in the original clinical trials, the rate of such side effects was nearly the same as placebo. And I personally know about 10 men who take it and none have experienced side effects. I think people make these claims to get attention in forums, or it’s simply a placebo effect.
select * from changes(tablex)
The use of a streaming system adds a tremendous amount of complexity for no benefit, because the original data source is batch. A simpler solution would be to process the data in 5-minute batches and use a BigQuery MERGE statement to apply that "patch" to the existing table.
You really shouldn't use Kafka unless you have a true streaming data source and a hard latency requirement of < 5 minutes. It's much easier to work with data in small batches, using local disk and blob stores like S3 as your queue.
The editor is very different as well: Mathematica pioneered the notebook concept that was copied by Jupyter, and the Mathematica user experience is still much more comprehensive and tightly integrated than anything else.
For linear algebra, they both use Intel MKL under the hood and get exactly the same performance.
> How come you haven't go to 10T?
We don't see a lot of 10 TB single datasets "in the wild". We think 100 GB - 1 TB is more representative of the average real-world data warehouse user.
> for us the warmup time separates the time to compile and code gen
You should include compile time---your compile time is really good! Redshift sometimes takes longer to compile the query than to run it, though I think they may have improved this in the last 6 months.
1. Total run time is not an appropriate way to summarize the performance across queries, because some queries take 100x longer than others. The appropriate way to summarize this kind of data is to use the geomean [3].
2. The official TPC-DS queries make heavy use of grouping sets, which are a rarely-used SQL feature. I think TPC-DS is better if you rewrite the queries to eliminate grouping sets.
3. You used the exact same queries to "warm up" the data warehouse, and to test the performance. Some data warehouses (notably Redshift) aggressively cache intermediate compilation results, so they are much faster the second time they see a query or even a fragment of a query. To model a real user submitting queries interactively, you should use warmup queries that are similar to but not the same as the ones you use to measure performance.
4. You can solve the "vendor benchmarking their own product" problem by submitting a PR to our repo [4], which currently tests Redshift, Snowflake, BigQuery, Azure SQL DW, and Presto. We'd be happy to review it and endorse the timing if it meets our standards for fairness!
[1] https://fivetran.com/blog/warehouse-benchmark
[2] https://www.youtube.com/watch?v=XpaN-PqSczM
1. Periscope emphasizes directly writing SQL.
2. Looker hides the SQL behind "developer mode" and emphasizes clicking dimensions and measures.
3. Periscope supports R and Python somehow, though I haven't used it.