The comments can be misleading. About 2600 people thus far and a few bots have been able to view it. Sunday is the best day to see all the unusual browsers and applications people browse the web with. Weekdays tend to be more mainstream bog standard browsers. This is a good test.
Rejects me too. Not a bot and connecting directly with a "normal" old-fashioned up-to-date web browser. How will you evaluate how many humans vs bots you blocked?
I used to log the bots too but there are so many they drown out the humans. I fully expect to block a lot of real humans as many people today use VPN's and most VPN's will egress from a data-center. Maybe it would make sense to break this out into a blog and a demo blocking site some day.
As of now roughly 4385 real humans can reach it. 9 people can not reach it. A few of those 9 I expect to be blocked being on VPN's that egress data-centers.
the top tier ai scaper and bot defeat companies, they are big business. really they are not grey market at all.
the real humans you are seeing, are bots.
brightdata is one vendor, they have 150m genuine residential ips, aquired through standard business.
actually, since this was kind of an invitation, i scraped it. the volume is so small compared to the ai corpus, this volume does not cost anything. as i suspected it was a 100% success rate. the only page it did not reach was the one needing authentication. the tests, they were fine. of course, ai knows ascii from this type of data but it wouldn't be able to read the page unless you use a screenshot.
the bots are not interested in the headers. they never see them. imagine a normal person at home using their browser, that's the bot. the reason this is the case is because the bot ip is someone's house (at least for premium proxy), and the browser fingerprint is a recording of someone's browser. oversimplified of course.
the 'bots' are the public facing corpo scrapers, and the 'users' are bots run by data vendors.
to test this, there is a ranking on 'foil' bot detection. to actually detect them they need 0.05 usd per check, and they are more interested in fraud via automated interaction and account creation.
grab a free account, you can very easily collect such data.
The comments can be misleading. About 2600 people thus far and a few bots have been able to view it. Sunday is the best day to see all the unusual browsers and applications people browse the web with. Weekdays tend to be more mainstream bog standard browsers. This is a good test.
Rejects me too. Not a bot and connecting directly with a "normal" old-fashioned up-to-date web browser. How will you evaluate how many humans vs bots you blocked?
I used to log the bots too but there are so many they drown out the humans. I fully expect to block a lot of real humans as many people today use VPN's and most VPN's will egress from a data-center. Maybe it would make sense to break this out into a blog and a demo blocking site some day.
As of now roughly 4385 real humans can reach it. 9 people can not reach it. A few of those 9 I expect to be blocked being on VPN's that egress data-centers.
the top tier ai scaper and bot defeat companies, they are big business. really they are not grey market at all.
the real humans you are seeing, are bots.
brightdata is one vendor, they have 150m genuine residential ips, aquired through standard business. actually, since this was kind of an invitation, i scraped it. the volume is so small compared to the ai corpus, this volume does not cost anything. as i suspected it was a 100% success rate. the only page it did not reach was the one needing authentication. the tests, they were fine. of course, ai knows ascii from this type of data but it wouldn't be able to read the page unless you use a screenshot.
the bots are not interested in the headers. they never see them. imagine a normal person at home using their browser, that's the bot. the reason this is the case is because the bot ip is someone's house (at least for premium proxy), and the browser fingerprint is a recording of someone's browser. oversimplified of course.
the 'bots' are the public facing corpo scrapers, and the 'users' are bots run by data vendors.
to test this, there is a ranking on 'foil' bot detection. to actually detect them they need 0.05 usd per check, and they are more interested in fraud via automated interaction and account creation.
grab a free account, you can very easily collect such data.