← Back to context

Comment by jkarneges

5 days ago

I've had no trouble hitting the Firebase API at the speed items are created, with a 5 second delay between retries.

For scraping HN directly, in my experience you have to go extremely slow, like 1 minute between fetching items. And if you get blocked, it may be better to wait a long time (minutes) before trying again rather than exponential backoff, in order to get out of the penalty box. You'll need a cache for sure.