Comment by felarof

2 days ago

We would've preferred to build this as browser extension too.

But we strongly believe that for building a good agent co-pilot we need bunch of changes at Chromium C++ code level. For example, chromium has a accessibility tree for every website, but doesn't expose it as an API to chrome extension. Having access to accessibility tree would greatly improve agent execution.

We are also building bunch of changes in C++ for agents to interact with websites -- functions like click, elements with indexes. You can inject JS for doing this but it is 20-40X slower.

How is that accessibility tree different from the “accessibility snapshot” that you can get from Playwright for example?

I was tackling a similar problem few weeks ago and I found that playwright MCP was the most usable solution in my case. It doesn’t use an extension but it debugs the browser tabs (I guess using dev tools protocol) but I agree the experience was suboptimal

Could you upstream that change in order to make it an extension in the future? I think people would not value it any less.

  • We don't mind upstreaming. But I don't think Google Chrome/Chromium wants to expose it as an API chrome extensions, if not they would've done this long time ago.

    From Google's perspective, extension are meant to be lightweight applications, with restricted access.

    • I'm not really interested in AI agents for my webbrowser, but it would be pretty cool to see a fork of chromium available that, aside from being de-googled, relaxes all the "restricted access" to make it more fun to modify and customize the way you guys are. Just a thought, may be more of a market for the framework more than the product :)

      See Sciter. A very cool, super lightweight alternative to Electron, but unfortunately it seems like a single developer project and I could never get any of the examples to run.

      https://sciter.com/

      5 replies →

I mean you could build the agent with a first principles understanding of the DOM instead of just hacking together with the accessibility tree