Comment by Kim_Bruning

Either way, Agents have a very different umwelt from humans. They don't 'see' the internet the same way we do. Where we see obstacles, they might not notice anything, and where they run into barriers, we might just click right through.

If you're even a bit hacky yourself, you might not see the internet the same way yourself either. Consider little tricks like looking at urls and trying others that fit the pattern; or hitting view source in order to download a pesky image... etc etc.