Comment by crazymoka
10 months ago
Why do you need mlx? Like your blog post by you never explain why things need to be used.
Why isn't using localforge enough as it ties into models?
10 months ago
Why do you need mlx? Like your blog post by you never explain why things need to be used.
Why isn't using localforge enough as it ties into models?
I was just trying to make sure is maximally performant, and did it with MLX because i am running on mac hardware and wanted to be able to run 30b in reasonable time so it can actually autonomously code something. Otherwise there are many ways to do it!
It would be nice if you mentioned it's about apple silicon, and not apple intel computers. They're still ubiquitous nowadays
we're on the 4th generation of silicon now
I believe mlx will allow you to run the models marginally faster (per a recent blog post by @simonw)
Yeah, you don't necessarily need it but it's optimized for Apple Silicon and in my experience feels like it gives slightly better performance than GGUFs. I really need to formally measure that so I'm not just running on vibes!
I for one, am willing to just trust you bro ;)
1 reply →
mlx is an alternative model format to GGUF. It executes natively on apple silicon using Apple's AI accelerator, rather than through GGUF as a compute shader(!). It's faster and uses fewer resources on Apple devices.