For scraping HN directly, in my experience you have to go extremely slow, like 1 minute between fetching items. And if you get blocked, it may be better to wait a long time (minutes) before trying again rather than exponential backoff, in order to get out of the penalty box. You'll need a cache for sure.

  {
    "by" : "jkarneges",
    "id" : 45533018,
    "kids" : [ 45533616 ],
    "parent" : 45532549,
    "text" : "The HN&#x2F;Firebase API doesn&#x27;t make this easy. For <a href=\"https:&#x2F;&#x2F;hnstream.com\" rel=\"nofollow\">https:&#x2F;&#x2F;hnstream.com</a> I ended up   crawling items to find the article.",
    "time" : 1760043552,
    "type" : "comment"
  }