Hacker News Reader

Parent Full thread

nojito

Coolest part of clickhouse is it’s ability to do ETL automagically.

It really is a super power.

View on HN

What do you mean? Does it have an etl engine like MS SSIS or scheduler like airflow built-in?

I essentially load in two columns one called timestamp and another with a json blob.

I then use this to kick off materialized views to automagically pluck out relevant JSON fields into views

Similar to this

https://eng.uber.com/logging/

That's an interesting way to load data and then use with Materialized Views. However, I am curious how do you make efficient use of compression codecs[0] that Clickhouse provides, or some neat features like TTL policies [1] using this method?

[0]: https://clickhouse.com/docs/en/sql-reference/data-types/lowc...

[1]: https://clickhouse.com/docs/en/sql-reference/statements/alte...

Materialized views have a backing table where you can use the codecs. Can add ttl the ingestion table

The backing table in OP's case is just 2 columns, timestamp and a `String` column representing JSON blob. Can't do much there.

He said the injest table is defined as that not the MV backing table

Great minds think alike :)

We do the exact same thing at GraphJSON https://www.graphjson.com/guides/about

Reply on news.ycombinator.com