Comment by cyanregiment
6 hours ago
You’re a mad man - thank you!
Do I understand correctly that Ollama doesnt do that, and that’s why responses hang forever on a M3 running the same model through Ollama?
6 hours ago
You’re a mad man - thank you!
Do I understand correctly that Ollama doesnt do that, and that’s why responses hang forever on a M3 running the same model through Ollama?
Please don’t use Ollama.
https://sleepingrobots.com/dreams/stop-using-ollama/
Doesn't Ollama use llama.cpp so their point stands even if they used it directly?
It wouldn't be the first time ollama's llama.cpp fork reintroduced bugs and was missing important optimizations.
[dead]
Thank you!
afaik ollama relies on llama.cpp and mmap. mmap loads pages on demand and doesn't use the same explicit cache or parallel reads like my engine. Most likely ollama/llama.cpp will be way slower in this case