Comment by btilly
4 months ago
It would be more efficient for them to spin up a team to study this robots.txt thing. They've ignored that low hanging fruit, so they won't do the more sophisticated thing any time soon.
4 months ago
It would be more efficient for them to spin up a team to study this robots.txt thing. They've ignored that low hanging fruit, so they won't do the more sophisticated thing any time soon.
You can't make money out of studying robots.txt, but you can avoid costs skipping bad web sites.
Sounds like a benefit for the site owner. lol. It accomplished what they wanted.