Comment by rnhmjoj

3 days ago

Which companies are we talking about here? In my case the traffic was similar to what was reported here[1]: these are crawlers from Google, OpenAI, Amazon, etc. they are really idiotic in behaviour, but at least report themselves correctly.

[1]: https://pod.geraspora.de/posts/17342163

3 comments

rnhmjoj

Reply
nemothekid  3 days ago

OpenAI/Anthropic/Perplexity aren't the bad actors here. If they are, they are relatively simply to block - why would you implement an Anubis PoW MITM Proxy, when you could just simply block on UA?

I get the sense many of the bad actors are simply poor copycats that are poorly building LLMs and are scraping the entire web without a care in the world

  • rnhmjoj  3 days ago

    > why would you implement an Anubis PoW MITM Proxy, when you could just simply block on UA?

    That's in fact what I was asking: I've only seen traffic from these kind of companies and I've easily blocked them without an annoying PoW scheme.

    I have yet to see any of these bad actors and I'm interested in knowing who they actually are.