Ask HN: I keep wondering how embed.ly handles rate limits
Ask HN: I keep wondering how embed.ly handles rate limiting considering the number oof request they make to sites like youtube how come they are not blocked?
We have had conversations with the YouTube guys about this and they have asked us to use API's instead of directly scrapping pages.
Quora is currently rate limiting us a ton as well. We again have to directly reach out to them.
Most oEmbed endpoints aren't rate limited because they are generally not resource intensive. They are also meant to be used client side and therefore expect more traffic.
Embedly also does a fair amount of caching that reduces the number of calls that we have to make. This limits us, because we aren't a straight pass through, but helps us handle spikes of traffic.