Comment by syndeo
1 day ago
It’s actually really useful for web devs to have access to a local model. Whether or not browsers should bundle their own rather than using the system-provided model(s) is up for debate, however. For the time being, though, Google does have some of the better small ones.
Furthermore, users aren’t going to want to have to wait for an extra thing to download before their web apps can use AI.
That’s the thing… Without context of why, users probably wouldn’t want a 4 GB download. But they do want their web apps to work properly. When there’s a specific use case they’re interested in, they will want to have it, and they won’t want to wait.
You haven't even tried to provide a hypothetical example of what a web app should try to do using a local LLM, nor addressed the obvious questions about how that kind of thing should be standardized, what level of local LLM capability is reasonable for a web app to expect, or how permissions for that should be managed given that a local LLM is not just a tax on local storage capacity.
So why should anyone take it as a foregone conclusion that this is an instance where web devs should get what they want? In general, the browser should be acting in the best interests of the user and not automatically granting the wishes of every web site that wants to drain your battery.
One example I have that made me excited for this feature is the free recipe manager website I run.
Many of the paid-for competitors give users the ability to import unstructured recipe data these days from sites like instagram or at least text-only websites.
I can't afford to offer this as a feature since my website has no advertising and I just pay for it out of pocket, but it's an incredibly easy feature to add if you have the money to pay for tokens.
If I could use a local llm to do it though that runs in the person's own browser then I think it would definitely be valuable.
That said, I'm not sure the state of local llms provides a good enough experience yet (small models and slow) but that doesn't mean that in the future it might not be useful.
The propsosed apis do work for this purpose, albeit more slowly and lower quality
> not automatically granting the wishes of every web site that wants to drain your battery.
Pretty sure that ship sailed way back when Flash ruled the Internet, and it's still sailing more than ever today.
Browsers are just weird sandboxed VMs now. They have nothing to do with their original purpose. Don't be mad at me, I like shipping webapps that render documents server-side and use even JS incredibly sparingly. I'm just reporting what I see. The browser exists as a way to make developing completely proprietary apps with proprietary UIs for several platforms cheaper, and Chromium exists to help further that goal including, if necessary, being packaged up and shipped with those apps (Electron).
I for one run a small Scripture-study web app that makes use of this on Chrome browsers when available to provide summarizations of long commentary articles. I'm also looking to use it to power topical search.
I allow free open access to the content, as blocking it behind an account signup doesn't sit right ethically—it should be open and free for everyone.
The issue there is that there's no way to easily secure the API from being hammered by bad actors. (Due to the often-controversial response many have to Scripture, apps like these draw special kinds of negative attention.) You can set rate limits, but people can still abuse those, just to try to burn your money. I can get by for free (or relatively cheap, fully paid out-of-pocket as none of this is monetized) on Vercel/Netlify/etc, but inference is expensive, and a prime target for those trying to cause trouble.
If in the future the web exposes local "foundation" models that web apps can assume are present, that would open up great possibilities like these for indie devs like myself. Being able to offer useful and compelling features without worrying about abuse would be nice. That's my point.
There is a link elsewhere in this comment tree addressing all of that:
https://developer.chrome.com/docs/ai/prompt-api
Google's marketing for their latest new browser feature nobody asked for shouldn't be taken at face value. Somebody outside Google needs to provide a well-reasoned assessment of the feature proposal.
And having skimmed that page, it really doesn't answer most of the important questions. Are other browsers expected to ship Google's model, or put a different model behind the API that Google has documented as being specifically for Gemini Nano?
2 replies →
> really useful for web devs to have access to a local model
I’m not opposed to this. I don’t want Google, an advertising quasi monopoly, to be auto-installing its own AIs on everyone’s computers.
Totally fair. Depends on how it's used, really.
If it's to allow the web apps I use to work with more privacy, or to enable smaller/indie players (that can't easily afford to burn a bunch of API tokens for every user) to offer some basic AI-based features in their web apps, then I'm all for that.
This is the whole appeal of Apple's Foundation models on iOS too.