yingjunwu

it's production ready, and has already been adopted by:

* RisingWave (https://github.com/risingwavelabs/risingwave): SQL stream processing, analytics, and management. * Chroma (https://github.com/chroma-core/chroma): Embedding database for LLM apps. * SlateDB (https://github.com/slatedb/slatedb): A cloud native embedded storage engine built on object storage.

The database services are moving downmarket towards the developer segment. Stream processing should be democratized—the next-generation systems should be free, extremely easy to install, deploy, use, and maintain.

We introduced Await-Tree as a powerful tool for observability in Async Rust. Await-Tree is a backtrace tool designed natively for Async Rust, which allows developers to observe the execution status of each async task in real time and analyze the dependency blocking relationships between different futures or tasks.

Serverless is very appealing, but it's not a panacea for all your data infra problems.

Paradox in the data infra world: SQL's expressiveness might be limited compared to Java, yet SQL-centric data systems manages to generate more revenue.

We chose to build our own SQLSmith in Rust. It wasn't a mere whim. We tested various alternatives, analyzing their capabilities and adaptability to our needs. While some solutions showed promise, SQLSmith in Rust consistently emerged as the right solution.

I am the founder of RisingWave (http://risingwave.com/), an open-source SQL streaming database. I am happy to see the launch of Warpstream! I just reviewed the project and here's my personal opinion:

* Apache Kafka is undoubtedly the leading product in the streaming platform space. It offers a simple yet effective API that has become the golden standard. All streaming/messaging vendors need to adhere to Kafka protocol.

* The original Kafka only used local storage to store data, which can be extremely expensive if the data volume is large. That's why many people are advocating for the development of Kafka Tiered Storage (KIP-405: https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A...). To my best knowledge, there are at least five vendors selling Kafka or Kafka-compatible products with tiered storage support:

-- Confluent, which builds Kora, the 10X Kafka engine: https://www.confluent.io/10x-apache-kafka/;

-- Aiven, the open-source tiered storage Kafka (source code: https://github.com/Aiven-Open/tiered-storage-for-apache-kafk...);

-- Redpanda Data, which cuts your TCO by 6X (https://redpanda.com/platform-tco);

-- DataStax, which commercializes Apache Pulsar (https://pulsar.apache.org/);

-- StreamNative, which commercializes Apache Pulsar (https://pulsar.apache.org/).

* WarpStream claims to be "built directly on top of S3," which I believe is a very aggressive approach that has the potential to drastically reduce costs, even compared to tiered storage. The potential tradeoff is system performance, especially in terms of latency. As new technology, WarpStream brings novelty, and definitely it also needs to convince users that the service is robust and reliable.

* BYOC (Bring Your Own Cloud) is becoming the default option. Most of the vendors listed above offer BYOC, where data is stored in customers' cloud accounts, addressing concerns about data privacy and security.

I believe WarpStream is new technology to this market, and and would encourage the team to publish some detailed numbers to confirm its performance and efficiency!

It doesn't make sense to limit read rate purely due to `data scraping and system manipulation.` The underlying data infra is not scalable and cost-efficient.

Elon, adopt open-source technologies to reduce your GCP bill!

The modern real-time data stack has gained significant popularity in recent times. Systems such as Apache Pinot, Apache Druid, ClickHouse, RisingWave, and Apache Flink have been widely adopted in companies' data stacks. However, there are still some areas of confusion when it comes to this emerging space. During my conversations with customers, I frequently encounter the following questions:

* What is the difference between stream processing and real-time OLAP? * Why do I need stream processing if I'm already using an OLAP store? * Are RisingWave/Flink competitors to Pinot/Druid/ClickHouse?

These are excellent questions, and I enjoy personally explaining my perspective on the modern real-time data stack to each individual. However, in order to reach a wider audience interested in data engineering, I have written a blog post on this topic.

In summary, stream processing and real-time OLAP databases differ significantly in their design, implementation, and use cases. Stream processing is better suited for monitoring, alerting, and automation scenarios, while real-time OLAP is more suitable for interactive and exploratory analytics. Incorporating both types of systems in your data stack may be beneficial for your overall data processing needs!

Proud to see my name (https://twitter.com/YingjunWu) mentioned in Andy's blog. I was Andy's visiting PhD at CMU and was the top 1 contributor to Peloton (https://github.com/cmu-db/peloton).

Today, building a database from scratch is extremely difficult, for several reasons: 1. it anyways takes a long time; 2. there are so many successful (open-source) databases; 3. hiring top engineers are so expensive. 4. you won't get enough attention unless your system is drastically better than existing ones.

An interesting observation is that very few database was built since 2020 - almost all the newly built databases were developed on top of existing databases (PostgreSQL, ClickHouse, etc).

I started building RisingWave (https://github.com/risingwavelabs/risingwave) in early 2021. The only reason we built the system from scratch was that none of the existing systems can address the problem we are solving - distributed SQL stream processing at cloud scale. We tried Flink but gave up, as it's too heavy and it's architecture was not designed for the cloud environment.

If you want to build a database from scratch, or are simply interested in databases, we may talk.

Messaging system vendors: Confluent, Redpanda, StreamNative; Stream processing system vendors: Confluent, Decodable, Ververica, Aiven, RisingWave, Materialize; Python streaming vendors: Quix, Deephaven.

Who will win?

I am a founder of a VC-backed stream processing startup. Before that, I've been working on the stream processing domain for 10+ years. Recently, I wrote a new blog (https://www.risingwave-labs.com/blog/Rethinking_stream_proce...) to share my thoughts about stream processing. Combining my customer engagement experiences, I try to answer several key questions regarding stream processing: Why do we need stream processing? Why do we need a streaming database? Can stream processing really replace batch processing? I am still learning about stream processing, and any comments and suggestions are greatly appreciated!

I am the founder of RisingWave Labs, a startup building the next-gen SQL streaming database called RisingWave. We started building RisingWave in early 2021.

It's already 2023, and it means RisingWave will enter its 3rd year. Over the last two years, RisingWave has grown from a personal project to a project with 3.7K stars and 100+ contributors, and the startup has also come out of stealth and became a series-A company.

I summarized RisingWave's 2022 here in a blog: https://medium.com/@yingjunwu/risingwaves-2022-year-end-revi.... It was originally an internal blog circulated inside our company, but now I decide to make it public so that everyone can understand where we came from, where we are, and where we are going.

Any comment is welcome, and I wish everyone a great and successful year in 2023!

I'm the founder of RisingWave Labs. Glad to know that our company was mentioned in the blog.

Technology wise, agreed with Nikita that storage-compute-decoupling is the trend. Severless will also be the main focus.

Market wise, I observed several trends:

* streaming. streamlining the data pipeline is a new trend. RisingWave and Materialize are both addresssing this market. They will probably need to figure out how to compete against established solutions like KsqlDB and Apache Flink.

* real-time. There are probably 20+ vendors selling real-time OLAP databases. They all claim that they are cheaper, faster, better than Snowflake/Redshift. Just name a few: Druid, Pinot, ClickHouse, StarRocks etc. Let's see who will be the winner in 2023.

* small data. The biggest winner in this domain is Motherduck, a startup commercializing DuckDB. Redshift/Snowflake are a overkill for most companies who do not own large amount of data. Motherduck may be the right choice.

* hybrid. Several companies have started selling hybrid databases (mainly HTAP for now): Google AlloyDB, Snowflake Unistore, SingleStore, PingCAP TiDB+TiFlash, etc.

In 2023, we will also watch how Databricks challenges Snowflake in the data warehouse domain.

Best luck!