39 karma · joined November 2, 2022
a great way to abuse DuckDB
with pipy and uvx, it will be a bit easier for others to install the mcp server. Just to add a JSON, without need to download binary manually.
At the core, Timeplus is a streaming processor, powered by ClickHouse. So it can leverage thousands of powerful SQL functions from ClickHouse, including `file(file, 'CSV', columns)`.
We are late to the 1brc party, but might be the first as a streaming processor. Powered by ClickHouse, Timeplus can efficiently process both streaming data and historical data, just with SQL.
Feel free to try it yourself and share your number. The author got 39 second. I got 21 seconds. Maybe on M3 Pro, it can be less than 20 seconds. No tuning applied yet.
Proton is an open-source streaming processor, written in C++. We’re thrilled to introduce a major upgrade of Proton, which you can now read or write ClickHouse with streaming SQL. You can use Proton as a lightweight streaming processor to get live data from Kafka/Redpanda, apply simple or complex processing logic, then send data to your ClickHouse, no matter in local or in the cloud. You can also JOIN Kafka data with dimensional data in ClickHouse, without moving them to Proton.
I wrote this tutorial to share how to get started using Proton and ClickHouse together. Demo video included. Happy to discuss here.
ksqlDB is great for * SQL Interface * Real-Time Streaming Processing * Integration with Kafka * Stateful Processing * Scalability * Security
However, there are also limitations with ksqlDB: * Deep Coupling with Kafka * Heavy Resource Consumption * Not Designed for Analytics * Not True Open-Source
There are lots of similarities between Proton and ksqlDB. Both support: * Stateful streaming processing * Data persistence * Stream/table concept * Dual query mode, long running, unbounded push based query and bounded, pull base query * Query Kafka data and write analytic results back to Kafka
Most features of ksqlDB can be found in Proton, while Proton offers additional benefits: * Proton demonstrates stronger performance * Proton is more flexible when consuming Kafka data * Proton is purpose designed for analytic workloads * Support complex computing by JavaScript UDF (User Defined Function) * Proton is more developer friendly
To explore Proton yourself, visit the [Proton GitHub repo](https://github.com/timeplus-io/proton) or create your own workspace on [Timeplus Cloud](https://timeplus.com).
Just check pandoc installation and get started guide. Overwhelmingly detailed, like I never use command line before. Maybe it assumes many users of pandoc are writers or Information Developer, without much tech skills.
In short, data streaming is getting popular. Apache Flink, Apache Spark, ksqlDB are traditional players. They work well in some cases, but have challenges on easy-to-use, easy-to-deploy, on even performance sometimes
Proton, RisingWave, Materialize, etc are the alternatives. In the end, it's always 'case by case' to pick up the best tools working for your team in your projects.
If you think this is too general, well - Materialize no longer provide the latest code as an open-source software that you can download and try. It turned from a single binary design to cloud-only micro-service - RisingWave has been open-sourced for 1.5 years and released their cloud mid of last year. It builds its own row-based historical storage, while Proton leverages ClickHouse for faster OLAP-like workload. Also there are more SQL functions supported in Proton, because it's powered by ClickHouse. Proton uses local SSD as the main storage and has an option to send older data to object storage. While RisingWave primarily uses object storage, and use local disk as cache. So in general Proton can achieve lower latency for both streaming or historical queries.
I am a big fan of coffee and drink different kinds of coffee over time, even on the same day. It's all about the use cases and preferences. This could be applied to choose open-source tools to query live data. In the last Current conference, me and Gang had a talk about different tools and different coffee. You may check https://www.timeplus.com/post/query-kafka-with-sql-current23