Comment by ehhthing 7 months ago You say this like robots.txt doesn't exist. 2 comments ehhthing Reply toofy 7 months ago it almost sounds like they’re saying the contents of robots.txt shouldn’t matter… because google exists? or something?implying “robots.txt explicitly says i can’t scrape their site, well i want that data, so im directing my bot to take it anyway.” sitzkrieg 7 months ago so many things flat out ignore it in 2026 let's be real
toofy 7 months ago it almost sounds like they’re saying the contents of robots.txt shouldn’t matter… because google exists? or something?implying “robots.txt explicitly says i can’t scrape their site, well i want that data, so im directing my bot to take it anyway.”
it almost sounds like they’re saying the contents of robots.txt shouldn’t matter… because google exists? or something?
implying “robots.txt explicitly says i can’t scrape their site, well i want that data, so im directing my bot to take it anyway.”
so many things flat out ignore it in 2026 let's be real