Comment by cjonas
10 days ago
How do you get by when every major sites starts blocking headless browsers? A good example right now is Zillow, but I foresee a world where big chunks of the internet are behind captcha and bot detection
10 days ago
How do you get by when every major sites starts blocking headless browsers? A good example right now is Zillow, but I foresee a world where big chunks of the internet are behind captcha and bot detection
That's not really a problem for Stagehand. It's a problem for Selenium, Playwright, Puppeteer and others at the browser automation library level.
it's not really a problem for Playwrite, because Playwrite is really intended to be run by the owners of the website, not as a webscraper.
It may become a real problem for the usefulness of this style of LLM driven browsing.
Playwright does have some docs on scraping, which makes it seem like they do want to support that as a use case https://playwright.dev/docs/docker#crawling-and-scraping. I'm unfamiliar with that though and wouldn't be able to speak to addressing issues with scraping
Right, but I mean it's a plugin issue outside of Stagehand. That's currently how the space treats the issue.
There are plugins that extend the major browser automation libraries to do this sort of thing.
And undetectable browser automation doesn't use Playwright, Puppeteer, or Selenium at all.