HNHacker News
TopNewBestAskShowJobs

karakanb

742 karma · joined February 19, 2017

co-founder & ceo of Bruin (getbruin.com), building the next generation serverless data platform.
submissionscomments
karakanb··on Show HN: I built an open-source data pipeline tool in Go
while MySQL is very popular, it is very rare to see it in analytical/ML usecases, that's why we haven't added it yet. There's nothing from a technical POV that prevents us from adding, it just hasn't been a priority, I am happy to pull it up if that would help your usecases.
karakanb··on Show HN: I built an open-source data pipeline tool in Go
thanks! we already use DLT under the hood with ingestr, so some of our connectors already come from there. is that what you meant?
karakanb··on Show HN: I built an open-source data pipeline tool in Go
hey, thanks a ton for sharing your thoughts, I appreciate that!

I am sorry to hear that it didn't work, we do have a dedicated page for duckdb specifically here: https://bruin-data.github.io/bruin/platforms/duckdb.html

Would this help with it? I'd love to see how we can improve if you'd like to share your thoughts on that. Please feel free to join our slack community as well, we can talk directly there too.

karakanb··on Show HN: I built an open-source data pipeline tool in Go
hey, thanks a lot for sharing your thoughts.

I like the comparison page in Hamilton, and in their examples they operate in the asset level, whereas Bruin crosses the asset level into the orchestrator level as well, effectively bridging the gap there. What Bruin does is beyond a single asset that might be a group of functions, it is basically being able to build and run pipelines of that.

In terms of distributed execution, it is in our roadmap to support running distributed workloads as simple as possible, and Postgres as a pluggable queue backend is one of the options as well. Currently, Bruin is meant as a single-node CLI tool that will do the orchestration and the execution within the same machine.

karakanb··on Show HN: I built an open-source data pipeline tool in Go
love it, thanks!
karakanb··on Show HN: I built an open-source data pipeline tool in Go
glad to hear you like it, thanks!!
karakanb··on Show HN: I built an open-source data pipeline tool in Go
that's definitely coming, thanks!
karakanb··on Show HN: I built an open-source data pipeline tool in Go
hey, I didn't know that, definitely gonna take a look. thanks!
karakanb··on Show HN: I built an open-source data pipeline tool in Go
hey, absolutely. take a look at here: https://bruin-data.github.io/bruin/vscode-extension/overview...
karakanb··on Show HN: I built an open-source data pipeline tool in Go
I would love to add a dedicated section on this, and would love to learn a bit more from you in this. Do you have any particular example tools that you would compare Bruin in your mind that you would like to understand the difference better?
karakanb··on Show HN: I built an open-source data pipeline tool in Go
depends on what you mean by that, but we do use dlt through ingestr (https://github.com/bruin-data/ingestr), which is used inside Bruin CLI.
karakanb··on Show HN: I built an open-source data pipeline tool in Go
great point about the transformation pipeline, that's a very strong part of our motivation: it's never "just transformation", "just ingestion" or "just python", the value lies in being able to mix and match technologies.

as per the lateness: ingestr itself does the fetching itself, which means the moment you run it it will ingest the data right away, which means there's no latency there. in terms of loading files from S3 as an example, you can already define your own blob pattern, which would allow you to ingest only certain files that fit into your lateness criteria, would this fit?

in addition, we will implement the concept of a "sensor", which will allow you to wait until a certain condition is met, e.g. a table/file exists, or a certain query returns true, and continue the pipeline from there, which could also help your usecase.

feel free to join our slack community, happy to dig deeper into this and see what we can implement there.

karakanb··on Show HN: I built an open-source data pipeline tool in Go
I didn't know about Ray Data before, but just gave a quick look and it seems like a framework for ML workloads specifically?

Bruin is effectively going a layer above individual assets, and instead takes a declarative approach to the full pipeline, which could contain assets that are using Ray internally. In the end, think of Bruin as a full pipeline/orchestrator, which would contain one or more assets using various other technologies.

I hope this makes sense.

karakanb··on Show HN: I built an open-source data pipeline tool in Go
thanks!

I did look into CUE in the very early days of Bruin but ended up going with a more YAML-based configuration due to its support. I am not familiar with their flow package specifically, but I'll definitely take a deeper look. From a quick look, it seems like it could have replaced some of the orchestration code in Bruin to a certain extent.

One of the challenges, maybe specific to the data world, is that the userbase is familiar with a certain set of tools and patterns, such as SQL and Python, therefore introducing even a small variance into the mix is often adding friction, this was one of the reasons we didn't go with CUE at the time. I should definitely take another look though. thanks!

karakanb··on Show HN: I built an open-source data pipeline tool in Go
thanks a lot for the kind words, James!
karakanb··on Show HN: I built an open-source data pipeline tool in Go
hey, thanks for the shoutout!

I love the idea, effectively allowing going towards a direction where the right platform for the right job is used, and it is very much in line with where we are taking things towards. Another interesting project in that spirit is sqlframe: https://github.com/eakmanrq/sqlframe

karakanb··on Show HN: I built an open-source data pipeline tool in Go
good point, thanks. I'll definitely add some more details about the comparison between different tools.

I agree with you 100% on the language part, I think it is an interesting detail for a data tool to be built in Go, but we have a lot more than that, a couple of things we do there is:

- everything is local-first: native Python support, local VS Code extension, isolated local environments, etc

- very quick iteration speed: rendered queries, backfills, all running locally

- support for data ingestion, transformation, and quality, without leaving the framework, while also having the ability to extend it with Python

these are some of the improvements we focused on bringing into the workflows, I hope this explains our thinking a bit more.

karakanb··on Show HN: I built an open-source data pipeline tool in Go
great question! Meltano, if I am not wrong, only does data ingestion (Extract & Load), whereas we go further into the pipeline such as transformation with SQL and Python, ML pipelines, data quality, and more.

I guess a more comparable alternative would be Meltano + dbt + Great Expectations + Airflow (for Python stuff), whereas Bruin does all of them at once. In that sense, Bruin's alternative would be a stack rather than a single product.

Does that make sense?

karakanb··on Show HN: I built an open-source data pipeline tool in Go
Bruin orchestrates individual runs for single pipelines, which means you can use any tool to schedule the runs outside and the assets will be orchestrated by Bruin. You can use GitHub Actions, Airflow, a regular cronjob, or any other form of scheduling for that.
karakanb··on Tesla is looking to hire a team to remotely control its 'self-driving' robotaxis
Genuine question: instead of self-driving cars, why don't we have remote controlled cars all around? With cameras all around the visibility would not be as terrible, together with the current self-driving capabilities which can get through basic situation I would have imagined we'd see more of them around.
karakanb··on Notion's mid-life crisis
It still blows my mind that Google Docs still haven't taken over Notion's business altogether. I have used Notion for ~6 months for my own company, ended up going back to Google Docs because it was not worth it. I am sure there are a billion other uses of it, we weren't using any of its database-like features, just as an internal documentation platform, but still fascinating to see a business thrive on the lack of innovation of a giant.

In practice, Google could put a tree-like organization of docs on the sidebar, make the search a bit comfier, and make draggable blocks, and get 80% of the Notion users. I guess they don't have a financial incentive to make docs better, but I would gladly pay extra to have everything there.

I guess someone could build a browser extension that adds that UI to Google Docs, or eventually I'll go and do it myself.

karakanb··on A macOS and Windows open source alternative to ChatGPT
looks interesting, thanks for sharing.

What's the difference between this and the official ChatGPT app?

karakanb··on Show HN: I built an open-source data copy tool called ingestr
Hi there, thanks a lot for your comment and trying it out. Do you mind joining our Slack community via the link in the readme or create a github issue so that we can dive into this? I'd love to understand what doesn't work and provide fixes.
karakanb··on Show HN: I built an open-source data copy tool called ingestr
on it!
karakanb··on Show HN: I built an open-source data copy tool called ingestr
couldn't agree more! I see ingestr more as a common-scenario solution rather than a general solution that solves all cases, kinda like how I treat shell oneliners instead of writing an applicataion in another language. I guess there's space for both approaches.

I'll definitely take a look at CloudQuery, thanks a lot for sharing!

karakanb··on Show HN: I built an open-source data copy tool called ingestr
thanks a lot for your comment, glad to hear we converged on a similar idea! :)
karakanb··on Show HN: I built an open-source data copy tool called ingestr
it's on my roadmap for sure!
karakanb··on Show HN: I built an open-source data copy tool called ingestr
I have just released supporting local CSV file as a source in v0.1.3, let me know if this helps! :)
karakanb··on Show HN: I built an open-source data copy tool called ingestr
I don't think I invented anything tbh, I just relied on SQLAlchemy's URI formats, and I decided to abuse it slightly for even more config.
karakanb··on Show HN: I built an open-source data copy tool called ingestr
Also released local CSV file as a source in v0.1.3.
← PreviousPage 2 of 5Next →