Comment by jacquesm
2 hours ago
If Unsloth, DavidAU or mradermacher releases a quant that fits my hardware I'm definitely going to give this a shot.
2 hours ago
If Unsloth, DavidAU or mradermacher releases a quant that fits my hardware I'm definitely going to give this a shot.
They were given day zero access to release their quantizations[0]. The lowest quant will still require 256GB VRAM to run inferences, but as always I'm amazed by the Unsloth team by how much they can reduce the requirements for these powerful LLM's, and the easy-to-follow guides to get started with using them.
[0] https://unsloth.ai/docs/models/mistral-large-4
Ugh, ok that's out of my league :) Almost though, another 32GB and it would work.
Thanks for the link, appreciated!
https://huggingface.co/unsloth/Mistral-Large-4-1T-A52B-GGUF
I have no idea how you could shrink this down to fit on any type of prosumer hardware though. I doubt it's possible.
Yes, you are right, and I was too optimistic. I'd have to level up.
This is the stuff I've put together and it is clearly too small for this:
https://jacquesmattheij.com/local-inference-with-large-langu...