HNHacker News
TopNewBestAskShowJobs

shikhar

257 karma · joined June 26, 2010

Making durable streams serverless

shikhar@s2.dev

submissionscomments
shikhar··on Show HN: Cachey, a Read-Through Cache for S3
How we run it:

Auto-scaled Kubernetes deployments, one for each availability zone, currently on m*gd instances which give us local NVMe. The pods are able to easily push GiBps with 1-2 CPUs used — network is the bottleneck so we made it a scaling dimension (thanks KEDA).

On the client side, each gateway process uses kube.rs to watch ready endpoints in the same zone as itself, and frequently polls /stats exposed by Cachey for recent network throughput as a load signal.

To improve hit rates with key affinity, clients use rendezvous hashing for picking a node, with bounded load (https://arxiv.org/abs/1608.01350) – if a node exceeds a predetermined throughput limit, the next choice for the key is picked.

We may move towards consistent hashing – it would be a great problem to have, if we needed so many Cachey pods in a zone that O(n) hashing was meaningful overhead! An advantage with the current approach is it does not suffer from the cascaded overflow problem (https://arxiv.org/abs/1908.08762).

shikhar··on Why was Apache Kafka created?
Architecturally, there are a lot of the same considerations. A key difference is that we offer streams as a cloud API, and it does not have WarpStream's BYOC split – where some stuff runs in your environment and metadata lives in their cloud – so we can offer lower latencies. We are also not trying to be Kafka API compatible, S2 has its own REST API.

The dimensions we focus on are number of streams (unlimited, so you can do granular streams like per user or session), internet accessibility (you can generate finely-scoped access tokens that can be safely used from clients like CLIs or browsers), and soon also massive read fanout for feed-like use cases.

shikhar··on Why was Apache Kafka created?
> Jetstream clusters don't scale to many servers (they recommend max 3, I think)

Jetstream is even more limited than most Kafkas on number of streams https://github.com/nats-io/nats-server/discussions/5128#disc...

shikhar··on Why was Apache Kafka created?
You might like what we are building with https://s2.dev :)
shikhar··on Multi-Player Durable Stream Playground
S2 is a multi-tenant service and we multiplex granular streams for cost efficiency.

I believe the Prime Video migration was about compute...

shikhar··on Multi-Player Stream Playground
Open it in another browser or share the unique URL with someone else for a BBS experience.
shikhar··on Remote log implementation for XTDB using s2.dev
XTDB is an open-source bitemporal database

S2 is a serverless durable stream API

shikhar··on Scaling our observability platform by embracing wide events and replacing OTel
We have a customer using s2.dev for this capability – granular tail-able streams with granular access control (e.g. let an end user of a job tail it with a read-only access token). We'll be shipping an OTel endpoint soon to make it even easier.
shikhar··on Sync Engines Are the Future
We have had interest in using our serverless stream API (https://s2.dev/) to power sync engines. Very excited about these kinds of use cases, email in profile if anyone wants to chat.
shikhar··on Rust: Doubling Throughput with Continuous Profiling and Optimization
We were looking at the main branch of the sha2 crate at first, where the 'asm' feature has been removed (but this is unreleased, will be in 0.11.x) https://github.com/RustCrypto/hashes/blob/master/sha2/CHANGE...
shikhar··on Every System is a Log: Avoiding coordination in distributed applications
Indeed. We are trying to democratize that secret sauce. Since it is backed by object storage, the latencies are not what AWS enjoys with its internal Journal service, but we intend to get there with a NVMe-based tier later. In the meantime there is an existing large market for event streaming where a "truly serverless" (https://erikbern.com/2021/04/19/software-infrastructure-2.0-...) API has been missing.
shikhar··on Every System is a Log: Avoiding coordination in distributed applications
This is why we didn't actually call it logs as a service, but streams :P I meant to refer to the log abstraction this post talks about, see links therein. Observability events are but one kind of data you may want as a stream of durable records.
shikhar··on Every System is a Log: Avoiding coordination in distributed applications
This post makes a great case for how universal logs are in data systems. It was strange to me that there was no log-as-service with the qualities that make it suitable for building higher-level systems like durable execution: conditional appends (as called out by the post!), support very large numbers of logs, allow pushing high throughputs with strict ordering, and just generally provide a simple serverless experience like object storage. This led to https://s2.dev/ which is now available in preview.

It was interesting to learn how Restate links events for a key, with key-level logical logs multiplexed over partitioned physical logs. I imagine this is implemented with a leader per physical log, so you can consistently maintain an index. A log service supporting conditional appends allows such a leader to act like the log is local to it, despite offering replicated durability.

Leadership can be an important optimization for most systems, but shared logs also allow for multi-writer systems pretty easily. We blogged about this pattern https://s2.dev/blog/kv-store

shikhar··on Introducing S2
I failed to mention that we do want to support single-tenant cells for customers that need isolation.
shikhar··on Introducing S2
Fair :) Yes, we are pretty hyped about the possibilities with Iceberg, especially now with S3 Table buckets.
shikhar··on Introducing S2
(S2 Founder) Congrats on the success with Estuary! You are not the first person to tell me there is no/tiny market for this. Clearly _you_ thought there was something to it, when you looked to HN for validation. We may do a lot more on top of S2, like offering Kafka compatibility, but the core primitive matters. I have wanted it. It gets reinvented in all kinds of contexts and reused sub-optimally in the form of systems that have lost their soul, and that was enough for me to have this conviction and become a founder.

ED: I appreciate where you are coming from, and understand the challenges ahead. Thank you for the advice.

shikhar··on Introducing S2
(Founder) Thanks for the deep questions!

1) Storage is priced on uncompressed data. We don't currently compress segments.

2) It only affects chunk storage. We do have a 'Native' chunk store in mind, the sketch involves introducing NVMe disks (as a separate service the core depends on) - so we can offer under 5 millisecond end-to-end tail latencies.

3) The append ack latency and end-to-end latency with a tailing reader is largely equivalent for us since latest writes are in memory for a brief period after acknowledgment. If you try the CLI ping command (see GIF on landing page) from the same cloud region as us (AWS us-east-1 only currently), you'll see end-to-end and append ack latency as basically the same. TTFB for older data is ~ TTFB to get a segment data range from object storage, so it can be a few hundred milliseconds.

4) We have a deadline to free chunks, so we we PUT a tiny segment if we have to.

shikhar··on Introducing S2
(Founder) Unfortunately, if you access over a public IP, it is internet egress. Even if the client is in the same AWS cloud region. PrivateLink is the only option.
shikhar··on Introducing S2
(Founder) It's a number of dimensions. I get excited about the ordered throughput angle because I have personally cared about this in the past, and yeah a lot of folks may not need that :)

Simple API, reasonable pricing, latency flexibility, unlimited streams, _and_ elastic to high throughputs. All adding up to a great serverless experience.

Re: the data colocation. This is how most multi-tenant systems - including S3 itself AFAIU - operate. I understand there is a difference in level of trust vs a cloud provider, and the best we can do here while delivering a serverless experience is encrypting every single record at the edge of S2 where they transit in or out, with a tenant-specific key. We may even allow specifying it as part of the request, if clients want to manage the key for themself.

The best data security when leveraging any multi-tenant service is going to be client-side encryption, and we also want to make this super easy. With our planned Kafka layer, we plan on client-side encryption as a value add.

shikhar··on Introducing S2
(Founder) Thanks for sharing your thoughts. We are early and figuring things out. I agree egress cost is going to be a big concern. We want to do the best we can for users as we unlock some scale. During preview, we are focused on getting feedback so the service is free (we will need to talk if the usage is significant though).
shikhar··on Introducing S2
(Founder) An S2 stream is a durable log and can be replayed! We do want to add compaction support. Event sourcing is a great use case for S2.
shikhar··on Introducing S2
(Founder) Both S3 Express _One Zone_ appends and Azure's append blobs charge the regular PUT price for appends. It may work for you, but probably not if you want to do smaller writes.

Blob stores will also not let you do tailing reads, like you can with S2.

In AWS, S2's Express storage class takes care of writing to a quorum of 3 zonal buckets for regional durability.

I doubt object stores will go from operating at the level of blobs and byte ranges, to records and sequence numbers. But I could be wrong.

shikhar··on Introducing S2
(Founder) Yes we should create a video, thanks for the feedback.

In the meantime, checkout this quickstart which will have to streaming Star Wars with the S2 CLI and give you a pretty good sense of things https://s2.dev/docs/quickstart#get-started-with-the-cli

(You will have to apply to join the preview, but we are approving quickly)

shikhar··on Introducing S2
(Founder) We are not charging during preview. If anything, I wanted to be transparent about our planned pricing. Our mission is to make streams a cloud storage primitive, and I worked backwards from there in terms of our costs and expected costs looking ahead once we can scale a bit - based on concrete data points about what kind of discounts can be unlocked. I realized it was premature based on the comments here, so the price for internet egress has been updated. Thank you for your feedback.
shikhar··on Introducing S2
(Founder) I thought you were joking but coming back it could well be serious :)

When we say stream, we really mean The Log that Jay Kreps has a famous blog about https://engineering.linkedin.com/distributed-systems/log-wha...

We say stream because we would rather not be confused with "logs" as in application logs, but rather associate with the world of streaming data where this primitive is very relevant. We don't mean stream as in a TCP stream or live stream.

You can, however stream Star Wars on S2 ;-) https://s2.dev/docs/quickstart#get-started-with-the-cli

shikhar··on Introducing S2
(Founder) Yes, we will charge less for private connectivity. Pricing is transparent https://s2.dev/pricing - free during preview.
shikhar··on Introducing S2
(Founder) We plan to be multi-cloud over time. Kinesis has pretty low ordered throughput limit (i.e. at the level of a stream shard) of 1 MBps, if you need higher. S2 will be cheaper and faster than Kinesis with the Express storage class. S2 is also a more serverless pricing model - closer to S3 - than paying for stream shard hours.
shikhar··on Introducing S2
(Founder) That makes sense. We would eventually host the Kafka layer too - and will be able to avoid a hop by inlining our edge service logic in there.
shikhar··on Introducing S2
(Founder) No, we want to be in the same cloud regions as customers.
shikhar··on Introducing S2
(Founder) Thank you for the advice. I hope we can offer better when the deals come into play, but for now setting our planned internet egress price to $0.08/GiB.
← PreviousPage 2 of 4Next →