Hacker News Reader

Comment by alluro2 | Hacker News Reader

People like Geofabrik are why we can (sometimes) have nice things, and I'm very thankful for them.

Level of irresponsibility/cluelessness you can see from developers if you're hosting any kind of an API is astonishing, so downloads are not surprising at all...If someone, a couple of years back, told me things that I've now seen, I'd absolutely dismiss them as making stuff up and grossly exaggerating...

However, on the same token, it's sometimes really surprising how API developers rarely ever think in terms of multiples of things - it's very often just endpoints to do actions on single entities, even if nature of use-case is almost never on that level - so you have no other way than to send 700 requests to do "one action".

View on HN

> Level of irresponsibility/cluelessness you can see from developers if you're hosting any kind of an API is astonishing

This applies to anyone unskilled in a profession. I can assure you, we're not all out here hammering the shit out of any API we find.

With the accessibility of programming to just about anybody, and particularly now with "vibe-coding" it's going to happen.

Slap a 429 (Too Many Requests) in your response or something similar using a leaky-bucket algo and the junior dev/apprentice/vibe coder will soon learn what they're doing wrong.

- A senior backend dev

Thanks for the reply - I did not mean to rant, but, unfortunately, this is in context of a B2B service, and the other side are most commonly IT teams of customers.

There are, of course, both very capable and professional people, and also kind people who are keen to react / learn, but we've also had situations where 429s result in complaints to their management how our API "doesn't work", "is unreliable" and then demanding refunds / threatening legal action etc...

One example was sending 1.3M update requests a day to manage state of ~60 entities, that have a total of 3 possible relevant state transitions - a humble expectation would be several requests/day to update batches of entities.

> I did not mean to rant

Not at all, I sympathise, we're all like minded people here!

> One example was sending 1.3M update requests a day to manage state of ~60 entities, that have a total of 3 possible relevant state transitions

> but we've also had situations where 429s result in complaints to their management how our API "doesn't work", "is unreliable" and then demanding refunds / threatening legal action etc

That's painful, and at this point, we're beyond technical solutions, this need human solutions. If they can't realise that they're rate limited because they're basically abusing the API, they need to be told in no uncertain terms.

Of course I understand that it's not that simple, as a backend dev, my "customers" are usually other devs so I can be frank, but when dealing with B2B customers we often have to act like they're not in the wrong.

But that is a question that should be escalated to management right? If they charge the customer enough that allowing them to make 1.3M requests to update 60 entities makes sense, why not let them?

If they want the service stupidly overprovisioned to deal with these nutjobs, then that’s what we’ll do. I find that they’re generally receptive to the message of $10k per month to serve nutjobs, $100 per month to serve everyone else, though.

That’s the key - have that in place from the beginning.

Because many “enterprise” customers can spend literally millions doing shit the absolute wrong way, but have $0 budget for a developer to make it work right.

> Slap a 429 [...] will soon learn what they're doing wrong.

Oh how I wish this was true. We have customers sending 10-100s requests per second and they will complain if even just one gets 429. As in, they escalate to their enterprise account rep. I always tell them to buy the customer a "error handling for dummies" book but they never learn.

Heh, exactly my other reply - I feel for you, friend!

Another commenter said essentially the same thing, I sympathise, it's painful when the "customer" can't understand something clearly telling them they're doing it wrong.

I don't have an answer, but I wonder, for the sake of protecting your own org, is some sort of "abuse" policy the only approach; as in, deal with the human problem, be clear with them in writing somewhere that if they're seeing response X or Y (429 etc) that they're essentially abusing the API and need to behave.

The only thing that reliably works is to push the cost to the customer - so they can do whatever insanity they want, and they get charged accordingly.

And we’ve had cases where we had “fuckoff” charges in contracts (think $1 per API call after X thousand calls a day) and the customer just gladly pays tens of thousands of dollars and thanks us for fixing their problem.

The money is nice but sometimes you just want to shake them and say “look we have notifications you don’t need to poll the endpoint ten times a second, fuck it give us the code and we’ll fix it …”

I bet if the costs were an order of magnitude larger, they'd think the costs were as unreasonable as we think their huge number of requests are.

There's just no winning sometimes sigh.

It's hard to switch them over, but if you have it in the beginning, you boil the frog.

Yes

Well, if this is a supported (as in $) account, sure enough, have the API rate limits published and tell them in the most polite way to RTFM

I don't understand why features like S3's "downloader pays" isn't more widely used (and available outside AWS). Let the inefficient consumer bear their own cost.

Major downside is that this would exclude people without access to payment networks, but maybe you could still have a rate-limited free option.

They mention a single user downloading a 20GB file thousands of times on a single day, why not just rate limit the endpoint?

Their download service does not require authentication, and they are kind enough to be hesitant about blocking IPs (one IP could be half of a university campus, for example). So that leaves chasing around to find an individual culprit and hoping they'll be nice and fix it.

They could however rate-limit per IP with possibly a whitelist or higher limits for cooperative universities that are willing to prevent abuse from their own network. Unless they are facing a DDOS but then this appeal is even less likely to help.

Honestly, both sides could use a little more empathy: clients need to respect shared infrastructure, and API devs need to think more like their users

Reply on news.ycombinator.com