Comment by lemonish97
6 months ago
Never thought I'd run an LLM released in 2025, on my phone, in full BF16. With ~80tps on an iPhone 16 pro btw.
6 months ago
Never thought I'd run an LLM released in 2025, on my phone, in full BF16. With ~80tps on an iPhone 16 pro btw.
How do you actually run this on an iPhone?
With something like PocketPal AI (https://github.com/a-ghorbani/pocketpal-ai). I'd love hear HN'ers opinions on the "best" LM Studio-like option for iOS devices.
For iOS, OpenCat. Has iCloud sync, and one universal app for MacOS and iOS devices.
I use PocketPal. Can run any gguf model off hf.
you can also run it on Cactus - either in Cactus Chat from the App/Play Store or by using the Cactus framework to integrate it into your own app