WebSub: Open protocol for distributed pub–sub communication on the internet

For the kind of content sent over WebSub (generally an Atom or RSS feed with one or more long messages), it's much less than 50%.

The thundering herd happens when you have an incontrollable influx of traffic coming in, and you have no way to regulate it.

Think about what the hub has to do. It may have to notify millions of subscribers, deal with any errors, retry, etc. This is a very heavy duty messaging system that most publishers will not want to run themselves. And yet you want the publisher's domain name to be the well known resource that ultimately controls things.

Publishers may be blogs hosted on small websites or even things like cars, phones, laptops or home appliances that are not always online or have to work under tight resource constraints.

Publishers may wish to distribute their content through more than one hub. We don't even have to think of avoiding censorship to see why this increases availability.

I think making it possible to split the roles of publisher and distributor is a very good idea. You can still decide to implement both roles on one server.

WebSub is a protocol for people who want to implement the publish/subscribe pattern over HTTP callbacks (aka webhooks). Using webhooks means that subscribers don't need to have any kind of ongoing connection or session open to receive publishes. Subscribers are passive web servers and merely wait to receive an HTTP POST. No state, no connection, polling, or anything. The general model of HTTP callbacks is a simple scheme that's easy to implement using any programming language or platform out there, all of which have HTTP clients and servers capable of getting the job done with minimal fuss.

I have actually built custom systems that worked using a very similar pattern as this protocol, where clients of a service pass in URLs where they'd like to be notified when an event occurs. Perhaps this is why I find myself nodding along when I read the protocol spec. There wasn't any standard way to model this, so I just invented something on the fly. You also see this pattern implemented in services like AWS SNS's support for HTTP [1], in Google Cloud PubSub, Twilio, etc. Each of these has an entirely custom protocol for PubSub over HTTP callback, and not something that's standard. They all tackle similar issues like preventing attackers from creating unauthorized subscriptions to URLs, but in different ways.

WebSockets doesn't solve the same problem as WebSub. WebSockets require a continuous connection from a client to a service. An application will need to devise its own logic for resuming a session if the connection breaks.

WebSub requires no active connection nor session. WebSub subscriptions could remain functional for months at a time (really indefinitely), with there being no communications whatsoever between messages. The system that initiates the subscription can be different than the one that receives the publishes, which is valuable because it means that messages don't all have to go to a single place. Publishes are sent to the domain name specified in the subscription URL. The subscribed web server could change regularly and everything will work as long as the DNS name keeps pointing to the right place. You could use multiple web servers to handle the subscription, by putting multiple servers in the DNS record, or you could use a load balancer in the same way as other web requests. This means you can scale easily. These are the kinds of benefits you get from building subscriptions on top of HTTP. All of the standard techniques and standard software "just work".

I'm not an expert on XMPP, but I suspect XMPP would also be a bad fit for this use-case, and would also require continuous connectivity from the subscriber (please correct me if I'm wrong.) I think the same is true for MQTT but I'm not an expert on that either.

As a person who has built and used multiple systems following this general abstract pattern, I think this is a good attempt at drafting a standard protocol. My impression reading the spec is that its designers had a good idea what problem they wanted to solve, and what kind of characteristics they wanted the solution to have, and came up with a protocol that succeeded in meeting those requirements.

What's the objection to hub.secret? That facility doesn't seem essential to a minimal version of a protocol like this, but I understand why they included it. It provides a simple way for the subscriber to authenticate that the content they're receiving is legitimately the result of their subscription to the topic, and not e.g. an attacker's subscription, or an attacker system that's trying to impersonate the hub. How would you tackle this issue in a simpler way? (It would not be easy to solve this problem within the protocol using TLS, for example.)

[1] https://docs.aws.amazon.com/sns/latest/dg/SendMessageToHttp....

Funnily enough, the first drafts of this protocol (back then, called PubSubHubbub) were written circa 2008, so this specification is about a decade in the making.

At the time it was distributing content between a number of the bigger blogging/publishing platforms of the day, and also notifying search engines so they could update their indexes more quickly.

If anything it seems like the standardization process was too long and missed the boat here (this particular problem is now most often solved by proprietary protocols), rather than being "rushed through".

Can't deny that the world has changed a lot during the lifespan of this idea, though. Cellular-connected computers in our pockets were barely on the radar when this spec was first written. I'm sure some would argue that the burdens of publishing have now shifted on to the reader (probably battery powered, spotty connectivity) whereas in this spec's original universe the burdens were on the publisher (CDNs not yet as widespread, more independent publishing from web hotels, etc).