miked98

Does it require a semantic layer such as DBT, Lightdash, or Rill?

In our experience, the primary driver of Snowflake costs is not the compute for aggregation, but the compute required for lots of reads/scans.

We recently built a Snowflake-to-ClickHouse pipeline for a customer where aggregates are built hourly in Snowflake, then pushed into a ClickHouse table to power their user-facing dashboards.

By offloading dashboard queries to ClickHouse, they slashed their Snowflake bill by ~25%, which was worth millions to them.

(Admittedly, running aggregations elsewhere—for example, in Spark—could further reduce costs, but you would then need Iceberg to make the tables queryable in Snowflake.)

Yes thanks to DuckCon team it’s here:

https://youtu.be/_IqvrFWY7ZM?si=1ux9SGUsh4kDs-ff

Alongside several great talks including Rusty Conover presenting Airport - Arrow + DuckDB — and Christophe Blefari (Bl3f) introducing a new, lightweight orchestrator called yato.

Rill founder here, I have no comment on the UI similarity :) but I would emphasize our vision is building DuckDB-powered metrics layers and exploratory dashboards -- which we presented at DuckCon #6 last month, PDF below [1] -- and less on notebook style UIs like Hex and Jupyter.

Rill is fully open-source under the Apache license. [2]

[1] https://blobs.duckdb.org/events/duckcon6/mike-driscoll-rill-...

[2] https://github.com/rilldata/rill

Here were the initial responses from HackerNews in 2011 :)

“It’s always tempting to build it yourself. “

“They should have just used QlikView.”

“HANA has been doing this for at least 5 years now.”

via https://news.ycombinator.com/item?id=2501160

By DDG you mean the "Data Drinking Group", right? I've heard those guys are pretty active on HN.

D3.js powers all of the visualizations at Metamarkets, thanks to Vadim Ogievetsky, who got an early look at the D3 code base. It's a powerful framework that extends beyond visualizations: it can be used to attach data to any part of the DOM, not just SVG elements.

Here's one example that weaves together both data and graphics: http://labs.metamx.com/.

Practical, extensive, and timely piece on the nuts and bolts of weaving Hadoop and EC2.