HNHacker News
TopNewBestAskShowJobs

tayloramurphy

109 karma · joined January 31, 2021

AI PM @ Astronomer

Prev. Founder/CEO @ Arch.dev, Maintainer of Meltano, Data @ GitLab

Email me <firstname.lastname> @ <company domain>

submissionscomments
tayloramurphy··on Meltano: ELT for the DataOps era
Yep! We have this issue[0] open. I'm also interested in getting Prefect and GitLab CI as orchestrators too.

[0] https://gitlab.com/meltano/meltano/-/issues/2393

tayloramurphy··on Meltano: ELT for the DataOps era
I'm not aware of any. I did just open this issue[0] in the Meltano project to open discussion with the team/community. It could be an interesting iteration on the Singer Spec[1] if we find that users are interested in it and it helps solve some bottleneck challenges.

[0] https://gitlab.com/meltano/meltano/-/issues/2616 [1] https://github.com/singer-io/getting-started/blob/master/doc...

tayloramurphy··on Meltano: ELT for the DataOps era
It's dependent on your use case, but the three examples you've listed here all have a slightly different approach to the market. Stitch (as mentioned in a comment below) is SaaS only. Pipelinewise is open source but as far as I know have no plans to build a company around it. With Meltano, we're aiming to grow the project and community and eventually build a business around it in a similar manner to what GitLab has done. Our docs[0] have more information about our current focus and roadmap if you're curious.

[0] https://meltano.com/docs/#focus

tayloramurphy··on Meltano: ELT for the DataOps era
You're correct, it currently does integrate with Airflow and dbt. I'm a big fan of both having used them basically every day for the past 3 years. The dbt community has done wonders for me personally and professionally and we're hoping to help build upon the dbt community (and others) to include open source data integration.
tayloramurphy··on Meltano: ELT for the DataOps era
Yep - here's the link to the podcast [0]. And you're correct, it is a project with GitLab. For the past year or so there's been 1 person on the project, but there are now 2 (today is when I switch from the GitLab Data team to Meltano) and a 3rd will be joining us in a week!

[0] https://www.dataengineeringpodcast.com/meltano-data-integrat...

← PreviousPage 3 of 3