Cloudflare K2: serverless event streams
blog.cloudflare.com
blog.cloudflare.com
I am excited about this future. Give me stateless servers and a storage bucket over having to manage systems with disks any day.
I do wonder if we will see an expansion of the s3 api to support more of these use cases. S3 added a janky file append operation to their new express-one-zone bucket type, and limited to 10k total file append operations. I wonder what else we will get in the next few years.
I am currently using Cloudflare R2 right now and if you see their forums, there's always the occasional post about objects going missing.
Consider your cloud costs.
Backblaze's famous reports put HD failure rate is ~1.39%, so for the 11 9s you get as a guarantee from S3. Assuming nothing else fails, you'd need at least 6 independent copies to get that, plus all the effort to engineer recovery, and constant upkeep.
Suddenly, S3, even when considering bandwidth costs, seems like a steal.
If you have two sets of hardware the server can run on (cold standby), and RAID, and are competent at physical IT work, you can have faulty hardware replaced in an hour. Drive fails - replace it. Anything else fails - swap the drives to the other machine, boot it up and then troubleshoot the original.
Most likely you don't even need that. If the app server runs on a standard platform like Windows you can shuffle it over to some spare tower PC. You hear horror stories of dusty towers that nobody knows what they're doing - the horror there isn't from running server software on a tower, but from unmaintained servers no matter the form factor
Don't go for the "object gateway" compatibility layer - just use raw Ceph if you're writing your own app.
> I do wonder if we will see an expansion of the s3 api to support more of these use cases
This is actually an area where I think we have a big leg up on folks building on top of S3. My team (which built K2) sits next to the R2 team, and we have the opportunity to co-evolve the products in mutually beneficial ways.
I think the opportunity extends below 100ms too, particularly given the existence of faster object storage tiers like S3 express or more recently GCS rapid bucket (both only offering single-zone durability, so still need to do quorum writes to get region-level durability as with standard tiers).
One of the tensions of course is how long to linger before flushing to object storage - you have to trade off directly between latency and cost of your API ops for PUTs.
When building the serverless offering of s2.dev (which is in a similar space, full disclosure!), we designed around stateful backend processes capable of constantly flushing multi-tenant objects (i.e., containing records from many streams), allowing streams to offer low ack latencies (~50ms p99 from same region) without blowing up the unit economics.
Congrats on the launch btw!
An obvious disclaimer is that the word "enough" here is carrying quite the weight: I expect it to get better, and each has their own requirements. Do benchmark yourself and don't make expensive decision based on an HN comment.
[0] https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpo...
The range of things you can do with blob storage and a (very simple) auth model are surprisingly broad.
We recently replaced our Docker container registry with S3 using a tiny tool [1] we built in-house. I think that even with current capabilities, we can still model a lot services as a very thin layer over object storage.
Well because your provider assumes you are not using all your bandwidth constantly. Cloud bandwidth is only billed for you actually use
And if you have two specific endpoints you need to transfer data between at a high rate, you can get stupidly cheap cost per GB on a leased line in exchange for making all that commitment upfront.
Alternatively you can stand up your own object store services, but that's not something I would like to do.
Huh? We're talking about the same S3 right? At list pricing, 1TB is $23/TB to store for one month, and about $90/TB (plus request fees) to send it out to the internet.
While hard drive prices are roughly 3x what they were a year ago, the price per TB of a new hard disk averages around $30/TB -- assume 2x overhead for other hardware and extra space for parity etc, a disk pays for itself in less than 3 months and lasts 5 years or more.
If you assume it takes 1 month to download that 1TB (about 3Mb/s) that's $29.16/Mbps. When I first started buying internet transit in Europe back in 2008 I think I was paying under $10/month. It's now under $0.10/Mbps pretty much anywhere in the US or Europe at the big datacenters.
None of the "big" object storage services are cheap. They're somewhat reasonable if you only access the data from within the same region, but absolutely insane if you ever want to ship that data outside of that cloud vendor (or to another region etc). The pricing of storage and egress has not changed in a decade (I believe AWS last lowered the price of either in 2016) and in fact it costs even more now due to things like NAT Gateways etc.
It's definitely not "shockingly cheap". It's just cheaper than $80/TB of gp3 or $45/TB of st1, and while sc1 is $15/TB it has a baseline performance of only 12 MB/s. There's quite a few companies out there that have $6/TB/month object storage plans with similar performance, rising to about $15-$18/TB/month for SSD backed storage with far lower latency figures.
Whatever AWS is selling you - except Deep Archive - I'll figure out a way to sell you for half that price, if you want, and it'll still be 80% profit for me. Your only downside will be that I don't know what I'm doing so it might not be reliable - but neither is AWS.
At those prices, it only makes sense to use it for data that never leaves that providers cloud (which I assume, but cannot prove, is their intention).
We use R2 (not affiliated) which doesn't have egress fees.
This is known as Stockholm Syndrome.
So you need something to wrap the layout into something Docker understands. Because S3 is not a server, you have to construct the tarball on the fly as the image is pulled and stream it straight into docker load.
Even better, the OCI distribution protocol (which was Docker distribution until a few years ago) is not a static-blob-over-HTTP protocol! The blob bits are and you can route them to blob storage but you need a smart server for a few key bits of the protocol.
Back in the day I made proposals for distribution formats that didn't have these flaws and were more distributed (and previous proposals like AppC's discovery had similar ideas) but they were roundly rejected by the Docker people.
As long as you can run on a CSP, and can engineer around the high-ish latency (most business cases can), it's extremely expensive to try engineer around it.
Haven't tried it yet but looks nice for simpler K8s deployments
The AWS Aurora white paper was released almost 10 years go. Aurora was announced in 2014 and the paper released in 2017.
(Aurora is built on S3)
I'm working on something like this for my business, but the server is the bucket. When you shut it off, the VM data is stuffed into a bucket and restored when you want it back on.
Our use case is that we rent out VM's with GPUs in them (on-demand, no-reservation, billed by the minute), and the GPU rental is a lot more expensive than the disk rental. Right now, we don't have a cluster of storage in our DC, so we just delete the data.
We support API/cloud-init, so if they have regular workloads, they can just boot a VM with whatever they need.
It is then... get some storage (in progress) and let people pause their VM and pay less for disk when they don't need the GPU. Fully on-demand GPU compute. Perfect for agentic workloads where you want your agent to be able to spin up larger models on enterprise gear for when it needs it.
So that object storage can become the default even for regular line of business applications.
Queues are great when you have a unit of work that needs to be completed, retried, and tracked individually. For example, a shop might need to call a payment processor API that can fail or timeout, and retry it until it succeeds, while polling on the frontend for the state of that particular message. With a queue, you can insert a message tracking that payment, and have a queue processor that keeps getting sent it until it succeeds or has failed too many times.
In a queue each item is its own thing that's important to someone, and queues give you APIs to interact with that particular item.
K2 is for moving large volumes of data around. Pricing is per GB, not per message. Records are produced and consumed in bulk, and what matters is that all records are processed, but no one is querying the state of a particular record. K2 also supports multiple consumers for the same record, and long term retention. For example, all of your applications may emit events when things happen, and those events need to be read by an alerting system, a system that durably stores them, and a system that uses them to build ML features.
Event streams are for multiple consumers, and can offer different guarantees. As far as I can tell, K2 is designed to ensure all consumers receive all events at least once (it's unclear to me whether this means they're continuously storing all events from the stream origin, or if older events age out at some point, or are dropped when they've been consumed by all known consumers).
Other types of guarantees with streams might be "at-least-once", "at-most-once", and "exactly-once" delivery, for different needs. Redis streams used to be at-least-once but it looks like they support all 3 use cases now. Some relational DBMSes also have the option to replicate by streaming their transaction logs to all servers in the cluster so each node maintains its own understanding of the database state (though stale reads can also occur in some/all? DBMSes that replicate this way, when a server is queried before receiving an update)
A stream can have up to 100 subscriptions. Each one has its own cursor, so every subscription sees every event independently. Events aren't dropped once they're consumed; they stay until the stream's retention period runs out, so you can replay or add a new subscription later.
Within a subscription, any number of workers can consume, with up to 128 batches in flight at once. Each batch is a lease owned by exactly 1 worker, and if that worker polls again it gets the same batch back. When it acks, the batch is done and the cursor moves forward. It can also nack to hand the batch back right away or extend the lease if it needs more time. If it does neither, the lease expires after 5 minutes and the same records go to the next worker that polls, as a new batch. That's where the "at least once" comes from, a late ack from the original worker is just ignored.
This means actual usage is $0.08/GB in the simplest case (one consumer) but fan-out consumer strategies get very expensive very fast.
There was just more news about it than with other companies ( I think they let go about 2 k. People a year ago).
( Not saying it's good, just a little perception balance)
My bigger concern would be their increasing grip on a lot of the market, and eventually becoming a monopoly (or part of the big tech "duopoly")
At this point this has already effectively happened. The average person doesn't realize the extent of it, because the products Cloudflare builds are inherently transparent to the average consumer.
The question is less about "vibe coding" the product, and more about how their acquired teams function. For most products they have released so far, they are backed by a company they acqui-hired. They will usually rebrand the product and absorb the team.
You would not be surprised if this release was from a product for a startup company rather than a large enterprise company.
Cloudflare hired a bunch of folks for sure, but they also fired a lot of folks. What they are doing these days is building products by buying out entire companies, giving them nearly independent authority to build a product like they would build a company.
Also, the boundary between OLTP and OLAP is blurring every day.
For folks who want an off shelf version of this you may be interested in https://github.com/viggy28/streambed
(Disclaimer: I am one of the committers)
I'd be glad if they did it from that inspiration, but probably not :/
[1] https://www.arroyo.dev/blog/arroyo-is-joining-cloudflare/
Message-Queue-As-A-Service! (MQaaS!)
Related:
https://en.wikipedia.org/wiki/Message_queue
https://en.wikipedia.org/wiki/Message_loop_in_Microsoft_Wind...
As far as I know, Kafka itself already supports offloading some data to S3 for long-term storage.
Based on the articles—which I didn't fully grasp—I’m wondering if there are additional benefits mentioned, such as multi-region distribution (though I find it hard to imagine how that would be implemented).
We went with a simpler and more user friendly consume API, that also allows much higher levels of read parallelism (particularly important if you're using something like Workers, which parallelize well but aren't very powerful individually).
But we know many companies are invested in the Kafka ecosystem, and we want to provide an easy on (and if necessary, off) ramp for them.
As someone who enjoys writing Kafka streams applications I am also looking forward to the day you support the Kafka APIs.
Having a cost efficient fully serverless Kafka compatible service would be great, and something I think many businesses would find useful.
Great work!
Our Monolog is similar in spirit.
Instead of building Monolog on object storage (R2, S3), we built it on our Dip, thus achieving extreme low latency and parallelism for ingestion and consumption.
Our novel architecture enables scaling to infinite consumers without upfront partitions (no magic, different tradeoff). This plays nicely with Slyp's data locality and application architecture.
Monolog is built on Rust, is lightweight, and runs on mobiles and servers alike.
The original reason for building Monolog was comically outlandish, Kafka was slow and required JVM, Redpanda was eating too much RAM - 2GB min which is absurd for our use case. Redpanda was also consuming so much CPU that the disgusting CPU fan noise had us feel emotional pain.
We would very much like to build object storage on Dip, but we are currently preoccupied and hence don't have any immediate plans to build one in the short term. Hence, it is R2 for now.
Would have been useful for us if we'd known about it earlier.
Monolog's use case goes beyond resource usage, but Redpanda users coming across this would definitely find this helpful.
You need... | Use
-------------------------------------------------------|-------
“Make sure this job gets done” | Queue
retries / dead-letter handling | Queue
delayed jobs | Queue
distribute jobs among workers | Queue
“Record that this event happened” | K2
multiple independent systems reading the same events | K2
replay old events | K2
ordered event streams | K2
Kafka-like architecture | K2Compared to self-hosted or cloud-hosted Kafka (e.g., Amazon MSK or Confluent), K2 is much cheaper, and fully serverless. There are no clusters to manage or scale, and consistent performance even as you vastly increase the amount of data.
The main downside is produce (and end to end) latency is higher (around 1s p99) than systems that rely on local disk replication, like Kafka.
So it's great if you're trying to move a huge amount of data around, or for use cases where cost is more important than latency.
In our case BLOB and large document transfers were handled in parallel, merging them together through object storage is a highly appealing package. Great work!