Comment by zkmon 5 months ago I'm waiting for FP8 quant, preferably from Google. 2 comments zkmon Reply mdp2021 5 months ago If you accept the "ggml-org":https://huggingface.co/ggml-org/gemma-4-12B-it-GGUF/tree/mai...https://huggingface.co/ggml-org/gemma-4-12B-it-GGUF/blob/mai... zkmon 5 months ago Do they run well on vLLM?
mdp2021 5 months ago If you accept the "ggml-org":https://huggingface.co/ggml-org/gemma-4-12B-it-GGUF/tree/mai...https://huggingface.co/ggml-org/gemma-4-12B-it-GGUF/blob/mai... zkmon 5 months ago Do they run well on vLLM?
If you accept the "ggml-org":
https://huggingface.co/ggml-org/gemma-4-12B-it-GGUF/tree/mai...
https://huggingface.co/ggml-org/gemma-4-12B-it-GGUF/blob/mai...
Do they run well on vLLM?