Comment by Confiks
2 days ago
Get into a venv, and run:
> pip3 install git+https://github.com/ml-explore/mlx-lm.git
> ./venv/bin/mlx_lm.generate --model "$MODEL" --temp 1.0 --top-p 0.95 --top-k 64 --max-tokens 128000 --prompt "Hello world"
Where $MODEL is an unsloth model like:
- unsloth/gemma-4-E4B-it-UD-MLX-4bit
- unsloth/gemma-4-26b-a4b-it-UD-MLX-4bit
Thanks!