Yeah you have to produce in parallel and use batching, but it works well in practice. We’ve tested it up to 1GiB/s in throughout without issue
841 karma · joined April 18, 2019
Yeah you have to produce in parallel and use batching, but it works well in practice. We’ve tested it up to 1GiB/s in throughout without issue
I know that is super frustrating, I’m sorry. We’re just really early and unsure how to begin even discussing public pricing.
We are 100% committed to that free tier. We designed our entire architecture around making that free tier so cost effective that we could offer it for free.
We want to get to a point where we can have transparent pricing on our website, it will just take us some time to figure that out after speaking with initial customers. We would love to hear your feedback though. You won’t get handed off to a sales guy or anything, it’ll just be a direct conversation with me (CEO) and my cofounder Ryan (CTO)
On the read side, the architecture is such that you’ll have to pay for 1 GET request for every 4 MiB of data produced for each availability zone you run in. If you do the math on this, it is much cheaper than manually replicating data across zones and paying for interzone networking.
RE:deletes. Deleting files in S3 is free, it can just be a bit annoying to do but the WarpStream agents manage that automatically. It’s creating files that is expensive, but the WarpStream storage engine is designed to minimize this.
I will do a future blog post on how we keep S3 GET costs minimal, it’s difficult to explain in a HN comment on mobile. Feel free to shoot us an email at founders@warpstreamlabs.com or join our slack if you care for a more in depth explanation later!
Obviously there is state somewhere since the cluster is storing data, its just offloaded to S3 + our control plane, while still keeping all the data in the customer's cloud account.
FWIW we subject WarpStream to continuous chaos/fault injection in our integration tests and staging environment to verify correctness and liveness properties. I wouldn't say they're far from our mind, we've just made a big trade off around latency that we think will make sense for a lot of people.
None currently, if you need latency that low WarpStream is not a good fit and you should probably stick with something more traditional like a very well tuned Kafka cluster.
It gets pretty pricey though for large volumes of Data.
I'm not sure what you mean. Message persistence is a fundamental feature of Kafka that almost everyone using it relies on, its not some esoteric feature no one uses. We're each coming from our own network bias here, but in my experience a lot of people are really unhappy with the operational toil associated with running Kafka at scale in production.
That does not work currently because there is no way to configure the agent with a Minio URL. That's just an oversight on our part though, I'll get that working within a few days.
The cache is for reads, not writes. There is no cache for writes.
We built our own because it needed to behave in a very specific way to meet our cost/latency goals. Running a MinIO sidecar instance means that every agent would effectively have to download every file in its entirety which would not scale well and would be expensive. We also have a pretty hard and fast rule about keeping deploying WarpStream as simple as rolling out a single stateless binary.
It requires some form of object storage (S3, GCS, Azure blob storage, etc). You could use minio if you want to host the storage yourself, although I think I need to make a few changes to make that configurable in the agent.
For local development you can use an in-memory or local file storage, but not for "real" usage.
That's fair, that statement is probably significantly colored by my own personal network / work experience.
Cheers! We really want to drive the incremental cost (both in terms of $ and management overhead) per GiB down as low as possible so people can start using these systems for more use-cases.
Nope, we provide the same durability and ordering messages as Kafka, albeit with higher latency. We never acknowledge produce requests until they've been durably persisted in object storage (P99 ~400ms), so its the same guarantee as running a 3AZ Kafka cluster with fsync enabled.
Thanks for the kind words! I'll just add that tiered storage is not quite the same thing because it means you still have to manage local disks and replication carefully, however briefly. Please reach out with questions any time!
WarpStream's current offering is a hybrid BYOC approach. The customer runs the agents in their cloud account, and we manage the metadata store for them remotely. This keeps all the customer data in their cloud account and their S3 bucket where we can't see it or touch it. It does mean that the customer has to run the WarpStream agents themselves, but they're just stateless containers that are pretty easy to manage.
We considered using FoundationDB for our metadata store, but ended up not in the end. In order to make our free tier cost effective we really need to make our metadata store as efficient as possible for this specific use-case which required something a bit more custom. That said FoundationDB is a fantastic piece of technology. Best distributed database I've ever used, and I've used many :)
We don't rely on any kind of timestamp-based last write wins policy for concurrent writes. In fact the agents will never write a file to S3 with the same name more than once or overwrite an existing file.
It guarantees correct ordering inside a partition for all acknowledged messages regardless of which batch they originated from. We don't synchronize clocks, the agents call out to our cloud metadata store which runs a per-cluster metadata store that assigns offsets to messages at commit time. Effectively "committing" data involves two steps:
1. Write a file to S3 2. "Commit" that file to the metadata store which will then assign the partitions at commit time 3. Return the assigned partitions to the client
Yeah our P99 for producer latency is ~400ms right now.
1. Yeah, we mention at the end of the post the P99 produce latency is ~400ms. 2. MSK still charges you for networking to produce into the cluster and consumer out of it if follower fetch is not properly configured. Also, you still have to more or less manage a Kafka cluster (hot spotting, partition rebalancing, etc). In practice we think WarpStream will be much cheaper to use than MSK for almost all use-cases, and significantly easier to manage.
That said it does require a lot of metadata to orchestrate all the different concurrent operations over S3. We handle this with a custom metadata store that we run in our cloud control plane.