There are many purposes where HTTP is not suitable at all. Realtime audio/video comes to mind - UDP is commonly used here because even the TCP latency is too much. Anything where you need a persistent, bi-directional connection is not a good match either. Anything where you need to push from server to client. In short: Anything that doesn't fit into the request/response paradigm.
In fact, many consider HTTP in its current incarnation to not even be suitable for the interactivity that we expect of modern webapps.
But ofcourse that doesn't stop the truly enlightened. Hence we got abominations like "Comet", a persistent stream-socket emulated on top of an inherently request/response-based protocol. Or RSS, which is nothing more than Usenet done really, really wrong - all lessons unlearned.
Sorry, got a bit carried away. But you get the idea I guess.
Be careful when you get carried away.
I meant to say: They not only avoid HTTP - they even avoid TCP, too.
Well, the phrasing was not ideal, I'll give you that.
UDP does not guarantee packet delivery, thus HTTP cannot ride on top of UDP.
It would be pretty hard to make http work over something that guarantees transport of a packet but is not connection oriented.
Where Reliable_protocol_wrapper does retransmissions, acks, sequence numbering etc.
There's no reason why you couldn't use HTTP over UDP if you like, I'm sure it's most likely been done in the past - if only just for fun.
It turns out lots of ISPs and broken routers and terrible corporate proxies only allow HTTP. They do things like require HTTP headers, so they can check for whitelists/blacklists. It's stupid and it breaks TCP/IP in general, but most people only care about the internet so it just works for most people.
To me comet is a testament to our collective failure to demand proper tools from the "big guys" (Mozilla, Opera, Microsoft). The term "web 2.0" was coined roughly 7 years ago and even long before that it has been clear that some sort of WebSocket would be incredibly useful to have in a browser.
Yet no single browser vendor (to my knowledge) has impemented it today.
Imagine where we could stand if only Firefox had a true, non-standard Socket class already. People would be using it (socket for the fox, comet-hacks for the rest) and IE market share would probably be dropping faster than ever because all the fancy new apps work better elsewhere.
But I digress, this is mostly whining. You have a point; the comet hack, as ugly as it is, is justified by the lack of proper alternatives. But still, an abomination on so many levels...
You say "all lessons unlearned" implying that RSS is worse than previous incarnations of solving the same use case. This is a common opinion I really am railing against. RSS is EXTREMELY popular. Usenet is a dying architecture. It's a case where being worse in the "obvious" technical architecture allows you to be better in things that actually matter: ease of access, simplicity of use, lack of installing things and wide support. These are just some of the reasons RSS took off. Easier means more viral, because easier means a shorter viral loop.
Of course, I think RSS is better for syndication than notification, and would prefer more sites like twitter use Webhooks in addition to RSS.
Well, ubiquity != adequacy. I understood the original author's article as a technical recommendation. He basically suggests to use HTTP for everything because he thinks "it's technically good for everything".
He made broad claims about how any HTTP based protocol will magically scale by leveraging proxies, loadbalancers and other existing infrastructure, completely ignoring the fact that many applications just don't fit into the request/response paradigm in first place.
RSS is a great example for the power of the internet that enables us to "build on what we have" without waiting for some standards body or greater authority to get moving. But it is also a text-book counterexample to his scalability and "one size fits all" claims. Polling just doesn't scale for these things and technically it's a step back from Usenet, that had these problems sorted out already. We went back to square 1 with RSS and are now locked up there until we get a true WebSocket and worthwhile persistent storage in browsers.
As an example where HTTP doesn't work so well, is pushing information as opposed to servicing requests. COMET is a hack and requires you to be very careful with clients. (e.g. the 2 connection limit in older browsers) You'd only really want to use it where there's no alternative, like in a browser.
Basically, HTTP is great for servicing individual requests. It can do kind of do persistent connections, but they're more an optimisation than part of the design and therefore optional for the client even if the server suggests them.
If your use case falls outside of the basic request/response premise, you can maybe use it for prototyping for a while, but you should probably switch to something more suitable to your setup after that unless you need to use HTTP for reasons outside of your control.
Sensor networks are a bunch of sensors that collect info, then network themselves to transfer what they know, usually to a mothership base station of some sort. Because it's a pain in the ass go out in the field and replace the batteries on these sensors, you have to make their transmissions energy-efficient. There is nothing in the HTTP protocol that takes energy consumption into account. Nor should it. It's not an application layer problem. However, the only reason I mention it, is that I've seen people start mixing the protocol layers together in a single protocol to achieve the energy consumption characteristics they want, which included the application layer.
So hence, that's one instance where HTTP isn't a first choice. As long as you know what HTTP is for, and don't use it blindly, you'll find that it probably suits your needs.