Hacker News Reader

Comment by skylan_q | Hacker News Reader

Sticking to the theme about gripes about APIs, why do so many companies have APIs and no webhooks? Polling sucks. Many SaaS providers will say it's "hard" to implement webhooks but they have an API available. I'd like to know what's so hard about making webhooks available.

I've developed a system that's supported webhooks and it was actually easier to create webhooks (and have users submit endpoints on specific events) than it was to create a proper public API for the same service.

View on HN

"deliver exactly once" is still an unsolved problem in computer science, so any webhook service is left with the options:

1.) only attempt once

2.) attempt x number of times

3.) attempt until delivery is confirmed

All of these can be done, but there are nuances and complications. It's not trivial, which is why we don't see more of it.

Also, and perhaps most obvious, if they force you to poll, they are putting the burden of development on you, whereas if they offer webhooks, they have to take on the cost of development themselves.

The difficulty of a foolproof solution isn't a great argument against a solution. That is, many cases, I'd rather have a webhook with the "only attempt once" strategy than an API I have to poll. "Pretty reliable" is often good enough. I think OP's question is valid.

Whether pretty reliable is good enough depends very much on what you are trying to achieve. I'd rather my salary wasn't deposited into my bank account on a pretty reliable basis.

I'd be okay with "at least once" :)

The reality of 'web hooks' in a really well written API is something that still cannot be relied on.

I use an API for a shipping company that fulfils orders for USA/Canada. It does have 'web hooks' to update orders with tracking numbers and status updates. It also has some type of regular update to make sure that any orders left 'open' are updated even if they should have been updated already.

It all works, but only 99% of the time... There are many orders left 'open' with the only option being to look up the orders and assess them manually. This is a big task even with only 1% or so to follow up and double check.

So, despite the wonders of the 'web hooks' approach, we are going back to API polling, to get all order updates since [last poll time] and updating order statuses accordingly. We will disable the web hooks once the straight API + polling approach is fully tested. A client library for the API will also have to be built so that simple getters and setters can be used for updating order statuses and inventory levels.

The 'web hooks' approach ultimately does not work reliably enough for us, so a more thorough and robust solution is needed. Really it would have been better to have not gone the 'web hooks' way at all, it would have spared us the problems we have had with only 99% of the information coming through. Nowadays customers are quick to talk online about poor service so this type of thing is of consequence.

Off the top of my head:

  - Handling failure
  - Handling retries
  - DDoS prevention
  - Guaranteed ordering at scale (while handling failure and retries)
  - Handling stale endpoints

...and several other issues. It's not that none of these are insurmountable, but there is a certain level of policy and engineering effort that goes into a web hook system, especially if it's expected to operate at any sort of scale. I learned a lot watching the folks at GitHub build and maintain the web hook system while I was there. It's not as easy as it seems.

Many API developers don't see much value in having realtime updates. As deeper backend server developers, they don't face the same level of urgency that frontend/app developers do (where the trendiness of realtime/reactive/etc has gone through the roof). Well, that and lack of tooling. Push servers/frameworks are a dime a dozen but not many are usable in an API context. So even when API developers do want to implement this, they hit a wall pretty quickly.

Shameless plug: our company (https://fanout.io) was created to solve exactly this problem. If you're working with an API that you wish had realtime push, please tell its developers about us. :)

You can make your own webhook system, but it will be totally nonstandard and custom, I believe.

RSS and ATOM are standardized mechanisms for conveying an event stream, understood by thousands of clients. I'm not aware of anything else similar.

by webhooks do you just mean a websocket API?

I think he means an HTTP request to a configurable URL when some event occurs. Like having GitHub make a request to a URL on your CI server every time it sees a push or a merged PR, etc.

Reply on news.ycombinator.com