Hacker News Reader

Comment by sixo | Hacker News Reader

Custom pipes mostly coordinated by Luigi (heavily customized) -> Redshift -> DBT -> Looker.

Some spark mixed in at the ingestion and transformation phases.

Like someone else said in this thread, we're currently battling Redshift scaling problems and are looking to offload more of the raw data to S3 using Spark to create read views.

No data catalog right now but the Looker setup is extremely well-maintained. Hoping to deploy Lyft's Amundsen to fix this more generally.

View on HN

Since your Looker setup is well-maintained, did you consider adopting the data catalog in Looker, what would be the challenge doing it?

The main reason is that only a subset of all our datasets are actually exposed in Looker, and Amundsen would have applicability to other data sources - certain production semi-OLAP DBs, the future hive-on-S3 deployment... that we have no plans to expose in Looker at all.

That said I haven't looked closely at how far we could go with Looker, I'll check it out.

I see. We actually develop a BI tool similar to Looker but also want to implement a data catalog feature embedded into it. I'm familiar with Amundsen but I believe that this kind of tool needs to have tight integration with the BI tool and Amundsen still has a way in terms of integrations. (searching among saved reports, dashboards and collaboration with teams, etc.). Otherwise, using spreadsheets is still easier and maintainable.

I would love to hear the features that you would need in a data catalog system.

Reply on news.ycombinator.com