amath

Congrats on the release. Can dstack work with kubernetes? Like say another team owns the infrastructure and I’m required to use k8s, is it interoperable at all?

This seems like a good model for sustaining open source, but raises some questions.

Does anybody know how the DuckDB foundation works? The sponsors are MotherDuck, Voltron, and Posit, which are heavily venture-funded. Do DuckDB Lab employees work on sponsored projects for the foundation?

I am also curious if anyone can shed light on what kind of contract work DuckDB does to align its work with the open source project. This has always seemed like the holy grail, but it is difficult to do in practice.

This is just that authors opinion. It seems clear that there is a spectrum of what users want from basic retrieval to content generation.

Sounds like you are writing a pipeline to enrich a stream of data with historical data from Snowflake. This is a fairly common pattern. If the data in snowflake is not changing often, you would want to cache that somewhere for use with the stream processor to avoid that query overhead and speed things up.

Most of those listed can meet your first 2 requirements. Looking further down the list, your requirement of SQL and a DAG type of representation will limit the list to only a few. I don't know if many of those listed provide both of those capabilities.

If you relax the SQL constraint, more of them are applicable like Bytewax and Kafka-streams.

Developer Voices podcast is one of my favorites.

More interested in the opinions of those on this site and I thought the article would prompt some interesting discussion.

Nice work on the performance boost :).

How does it compare with things like: 1. https://github.com/bytewax/bytewax 2. https://github.com/pathwaycom/pathway

I recently read this article (https://materializedview.io/p/from-samza-to-flink-a-decade-o...) about Flink and it commented on Flink grew to fit all of these different use cases (applications, analytics and ETL) with disjoint requirements that Confluent built kafka-streams, ksql and connector for. What of those would you say Arroyo is better suited for?

Maybe this is a stupid question, but how would airbnb be looked at using this as precedent. If I sign up for airbnb and use their automatic price optimization engine would this fall under the same algorithmic collusion?

Great article! The breakdown of ETL, Analytical and Production workloads is so critical to understanding the pieces of the ecosystem and the history from your experience makes it much easier to understand.

I'm always curious about who writes all the comments on Hacker News. I am more of a lurker than a commenter.

We built this live Grafana Dashboard that uses Bytewax and Proton to pull data from the Hacker News API, process it as a stream, and then present it in a live updating dashboard.

I can't tell if you think that's a bad thing. I see it often, but yah this is a whole tutorial in the README.

Ah, this had me laughing.

Maybe this is a contradictory thought, but this is probably a good thing. I think you could argue that most drug mules are doing it because they don't, or don't think, they have another option to change their current life. If this reduces the number of humans used to smuggle drugs, it is probably a good thing.

These types of stories are what drove me to software in the first place. The ideal of a group of people working towards the same goal, for sometimes different reasons, but not for the sole purpose of monetary reward.

Good luck with the project!

I also echo some of the others, it would be great to see a sort of comparison with the alternatives so that we can better understand how the project fits in.

I worked at GitHub and this is exactly what I always wanted as someone working with data and ML artifacts. We even prototyped something with GitLFS once that looked like a very hacky and less well-thought-out version of this. Great work.

I am often confused on how things like imperative stream processing frameworks are supposed to work with the streaming database, if at all. Like what are you supposed to interact with the streaming database with? Is it only for streaming ETL? Do you need another application layer that can natively integrate, orchestrate and function. Maybe using a stream processing framework with a streaming database could facilitate a full application stack.

Thanks for sharing! It does seem like a lot of trouble for little if any benefit.

That paper is too good not to link - https://www.usenix.org/system/files/conference/hotos15/hotos...

I hope more companies can be built around timely-dataflow.

Correct, there is the open source project and then the platform is proprietary and has additional features.

We are licensing those portions and those are what are requested to be in escrow. Do you mean a copy left or business source type of license?

Thanks for sharing. I hadn’t heard of many scenarios in recent years, but I’m not sure if that’s just because nobody advertises that they had to put the source code in escrow.

Nice! I want to see about using that in https://github.com/bytewax/stateful-chat to remove the OpenAI endpoint requirement.

Hey HN! I am one of the people working on Bytewax (https://github.com/bytewax/bytewax). Bytewax came out of our experience working with ML infrastructure at GitHub. We wanted to use Python because we could move fast, the team was very fluent in it, and the rest of our tooling was Python-native already. We didn't want to introduce JVM-based solutions into our stack because of the lack of experience and the friction we had trying to get Python-centric tooling working with existing solutions like Flink.

In our research, we found Timely Dataflow (https://timelydataflow.github.io/timely-dataflow/, https://news.ycombinator.com/item?id=24837031) and the Naiad project (https://www.microsoft.com/en-us/research/project/naiad/) as well as PyO3 (https://github.com/PyO3/pyo3) and we thought we found a match made in heaven :). Bytewax leverages both of these projects and builds on them to provide a clean API (at least we think so) and table stakes features like connectors, state recovery, and cloud-native scaling. It has been really cool to learn about the dataflow computation model, Rust, and how to wrangle the GIL with Rust and Python :P.

Would love to get your feedback :).

`pip install bytewax` to get started. We have a page of guides (https://www.bytewax.io/guides) with ready-to-run examples.

describing so many nightmares here :). The overhead of testing/running locally is so important and yet so frequently not optimized for.

Author of the post here. I would love to hear thoughts on folks experience using the different architectures and more pros and cons to each one. There was surprisingly not a lot of information comparing these different architectures that I could find and I would love to be able to update this post with more details.

https://github.com/rerun-io/rerun is such a great tool for creating interactive dashboards to look at real-time data. Curious if there are others looking for something like this and what the other use cases would be interesting. I have heard ideas like: - A system for watching debezium populate topics in kafka. - Quick exploratory telemetry dashboards.

Any others?

Cool, nice idea. Can you sub in different backend like bytewax (https://github.com/bytewax/bytewax) for stateful processing?

Have you tried Pulsar or Redpanda? Both seem mature enough and provide decent performance to probably meet your needs. What I hear is that Redpanda is a lot easier to manage than Kafka.

Yes, open to contributors!

I would say checkout PyO3 to learn more about how you can marry Python with Rust code. It is an awesome project.

Thank you!

Thank you! Would love to