Comment by davidsojevic
9 days ago
There's a fork of this that has some great improvements over to the top of the original and it is also actively maintained: https://github.com/lexiforest/curl-impersonate
There's also Python bindings for the fork for anyone who uses Python: https://github.com/lexiforest/curl_cffi
I suppose it does make sense that a "make curl look like a browser" program would get sponsored by "bypass bot detection" services...
Easy. Just make a small fragment shader to produce a token in your client. No bot is going to waste GPU resources to compile your shader.
Why do people even think this? Bots almost always just use headful instrumented browsers now. if a human sitting at a keyboard can load the content, so can a bot.
7 replies →
Can't they use a software renderer like swiftshader? You don't need to pass in an actual gpu through virtio or whatever.
6 replies →
You are just guessing, please stop. Also, you’re wrong. All serious scraping is using browsers today.
Can't a bot just collect a few real tokens and then send those instead of trying to run the shader?
3 replies →
There's also a module for fully integrating this with the Python requests library: https://github.com/el1s7/curl-adapter
All these "advanced" technologies that change faster than I can turn my neck, to make a simple request that looks like it was one of the "certified" big 3 web browsers, which will ironically tax the server less than a certified browser. Is this the nightmare dystopia I was warned about in the 90's? I wonder if anyone here can name the one company that is responsible for this despite positioning themselves as a good guy open source / hacker community contributor.