Comment by specialp
15 hours ago
I am dealing with the bots described in your first paragraph running a primary information source website. They are relentless and cost us thousands of dollars a month in wasted resources. They are almost certainly doing the bidding of much larger corporations as they are extremely sophisticated and seem to have infinite bots to send your way.
I am sure when they get unmasked all of the big tech companies contracting them will have plausible deniability
I'm not sure all companies will have deniability. It's pretty easy to see companies which are trying to evade detection or misusing their existing spiders for AI training purposes.
Some examples I've seen have crawlers which mask themselves 99% of the time by faking Chrome, but which will leak their real identity in UA strings which share the overall fingerprint of the swarm.
(I'm looking at you, Replit.)