paddybyers

I agree, no horizontally-scalable system is infinitely scalable in any dimension. You will hit some limit - for example, you might be able to support arbitrary scale in some dimension, but there is a bound on the sustainable rate of change, or the second-order rate of change, and you hit that limit. You seem to be suggesting though that hitting those limits came as a surprise, and/or there was no future architecture plan to address that when it happens, which I think is the real problem here. You would expect a team with the maturity of Github to understand what those limits are, and plan for them, based on predictable growth in demand.

I've run a disclosure program for ~7 years, which is an open paid program. However, over that time we've developed relationships with the most active and successful contributors, to the point that we'll now give them early access to new features to try out (all still paid for on the basis of rewards for problems found). This is proving especially valuable now in triaging the new deluge of noise from impactful issues.

I wrote about this this morning [1]:

> We're keeping our vulnerability disclosure program open - because even though they are rare, the genuine critical reports we receive, in amongst the noise, are still highly valuable. I don't think we're at the stage yet where finding those issues is a purely mechanical process; persistent, imaginative researchers still make a contribution to the process by finding things that LLMs by themselves, so far, haven't.

[1] https://www.linkedin.com/feed/update/urn:li:activity:7475447...

> I did get to the stage (not by conscious memorisation) of being able to assemble and disassemble Z80 code in my head, with some accuracy.

Same here.

I never got any fluency using EXX and the shadow registers - there were so few situations it was worth the effort. I always felt like I must be missing something.

I remember Jim Woodcock as really inspirational - he was working with my PhD supervisor in 1987. We were working on a variant of Z for specifying what, today, we would call CRDTs. I was also lucky enough to meet Tony Hoare the same year and discuss those concepts.

Who remembers the Occam folding editor (1983)? This was the first time I saw this kind of outline-based navigation for code: https://www.transputer.net/tn/03/tn03.html#x1-30002

Yes it's true. At Ably we support websockets, SSE and comet fallbacks (simple long-polling and streamed long-polling). It's less and less common but there are firewalls that fail to handle websockets correctly, or simply block them. I can't name specific companies/examples, but call centers are one example - the network and desktop environments are fully locked down.

We also see in these cases that streamed HTTP can also be broken by the firewall - for example a chunked response can be held back by the firewall and only forwarded to the client when the request ends, as a fixed-length response. Obviously that breaks SSE and means you can't just use streamed comet as a fallback when websockets don't work.

Disclaimer: I work for Ably. I agree that it's not valid to claim that websockets are the best fit in every situation. If the requirement is simply to stream server to client events, in an application where that doesn't depend at all on having any per-client state, then you can serve those things statelessly with SSE or HTTP/2. However, much of the time, the thing that makes websocket implementations complex isn't anything to do with websockets per se, but the fact that the server has per-client state; this state needs to exist somewhere in the backend, and it needs to survive reconnections (and so can't simply be some ephemeral session state in the specific server handling a connection). In this case, you could be using HTTP/2, or comet over HTTP1, or websockets, and the main technical issues you will face scaling to millions of users are going to be essentially the same. Websockets (at least, until web transport is here) just give you a more convenient and efficient primitive.

Disclaimer: I work for Ably. I agree in principle, so the libraries that handle websockets and also fallback transports using comet (eg SocketIO) are still widely used for that reason, and the commercial pub/sub service providers generally also support comet fallbacks. However, we now find that it is really very rare that clients are unable to use wss.

This ^ is my favourite writeup on the question of how you implement SOC2. I wish I had read that before we started - after going through the Type 1 and Type 2 process, we've ended up with the same conclusions. I've lost count of the number of times I've recommended that. Our experience (global b2b customers, heavily skewed to NA) is that SOC2 Type 2 is the most frequently requested/expected standard, and if you have that, not having ISO is very rarely a dealbreaker. Neither makes the security questionnaires go away; they continue to be mandatory, require expert input, and are a significant drain on time. However, having SOC2 and/or ISO does mean that you've already thought of the answers to the questions and you'll have a defensible position, backed up by a track record of independent audits, when your particular approach doesn't meet the "gold" standard implied by the questionnaire. (Edit: typo)

Ably CTO here. Well that went well ...

1) The realtime service and website are different things. The blog post is talking about the service, which has been continuously available.

2) Oops, the website fell over. We'll fix that. Thanks for all the advice :)

I lived there for 20 years and never noticed

We've been experimenting with Liftbridge at a range of scales and workloads, and clusters that are distributed across a single region perform very well. However, it's not really suitable for operating a cluster distributed across multiple regions because the latency adversely impacts the bandwidth of the Raft protocol that's used for cluster-wide coordination; this impacts the rate at which streams can be created, and the rate at which changes to In-Sync Replica sets (ISRs) can be propagated. As suggested, a federation of clusters, each covering a single region, would be required.

I can't speak for NATS streaming or Akka but it does look like liftbridge can fit that niche very well, depending on what features and semantics you need. Your services that publish can be assured that a message has been persisted to a quorum of replicas (strictly, the in-sync replicas) before receiving an acknowledgement, so you are able to assure that all onward processing to your downstream services occurs at least once for all acknowledged messages.

If you require fanout to your stream consumers then this is supported simply by having multiple consumers subscribe to each stream. If you require a competing consumers model then this isn't directly supported, but you can have load-balance groups which you could use to achieve something similar, but with a static configuration - that is, your consumer population can't resize dynamically, but you can distribute messages to a fixed set of streams, each of which has a consumer.

liftbridge doesn't yet support durable consumer-offset checkpointing, although this is advertised as being on the roadmap. In the meantime, if you wanted the consumer offset to survive consumer restarts, then the consumer itself would need to take care of persisting its current offset.