Comment by martinald

8GB RAM is not enough for a semi-decent model IMO. 12/16GB is better (4GB for model and 8GB for OS) and really if you were going hard on device you'd probably want more like 32GB (24GB for model + 8GB for everything else - you'd be able to run a 13b param model with larger context size with that).

Even still though people are used to the quality of huge frontier models, so it will feel like a massive downgrade on many tasks. The _big_ problem with all this is chained tool calling. It uses context SO quickly and context needs a lot of (V)RAM. This also completely undermines the privacy argument you make, because it will need to ask personal data if using OpenAI and put it in the prompt.

Yes I noticed Apple shipping higher RAM but it will take years for this to feed through to a sizeable userbase, and people are quickly getting ingrained to use an app like ChatGPT instead of OS level features. Even more so given what a flop Apple Intelligence 1.0 has been.

The key problem they've got is they've went hard on privacy (which means it is hard to square that with going all in on 3rd party APIs) but they've also been incredibly stingy with RAM historically, which really nerfs their on device options. Private compute is an interesting middle ground but their model options are incredibly limited currently.