Comment by cmeacham98
2 months ago
I work for Amazon, but not directly on web crawling.
Based on the internal information I have been able to gather, it is highly unlikely this is actually Amazon. Amazonbot is supposed to respect robots.txt and should always come from an Amazon-owned IP address (You can see verification steps here: https://developer.amazon.com/en/amazonbot).
I've forwarded this internally just in case there is some crazy internal team I'm not aware of pulling this stunt, but I would strongly suggest the author treats this traffic as malicious and lying about its user agent.
Randomly selected IPs from my logs show that 80% of them have the matching that forward confirming reverse DNS domain. The most aggressive ones were from the amazonbot domain.
Believe what you want though. Search for `xeiaso.net` in ticketing if you want proof.
So you said the IPs are residential IP, but their reverse DNS points to a amazonbot domain? Does that even make sense?
Reverse DNS doesn't mean much, they can set it to anything; can you forward match them to any amazon domain?
It's forward confirming reverse DNS. I assumed that everyone does that by default.
3 replies →