← Back to context

Comment by jgalt212

21 days ago

> Why do you need price trackers for airbnb?

More importantly, can Clawdbot even reliably access these sites? The last time I tried to build a hotel price scraper, the scraping was easy. Getting the page to load (and get around bot detection) was hard.

8 comments

jgalt212

Reply
kccqzy  21 days ago

That’s why the author explains that the page loads in a real Google Chrome instance on a real Mac mini from the same residential IP as his other devices.

  • jgalt212  20 days ago

    You do know that bot blockers keep track of metrics besides user agent and IP address? Hotels and concert ticket selling websites use some of the most aggressive bot blockers out there.

    • kccqzy  20 days ago

      I do. Most of these bot blockers block bots because of scale: these bots operate with superhuman speed and their traffic comes from all sorts of IP addresses. Tools like Anubis appear because such bot traffic dwarfs human traffic. And they typically have fake User-Agent headers set to a browser while their TLS/HTTP fingerprints would suggest they are made from curl or the requests library.

      This is different. There is no scale. The bot’s browsing session exactly replaces a human session. The browser is real.

      3 replies →