Comment by fibers
2 months ago
you have to feed it multiple arguments with rate limiting and long wait times. i am not sure if there have been recent updates other than the js interpreter but ive had to spin up a docker instance of a browser to feed it session cookies as well.
Yeah we had to roll through a bunch of proxy servers on top of all the other tricks you mentioned to reliably download at a decent pace
What are your thoughts on the load scrapers are putting on website operators?
What are your thoughts on the load website operators are putting on themselves to block scrapers?