Comment by dcreater
8 days ago
Called it.
It's very unfortunate that the local inference community has aggregated around Ollama when it's clear that's not their long term priority or strategy.
Its imperative we move away ASAP
8 days ago
Called it.
It's very unfortunate that the local inference community has aggregated around Ollama when it's clear that's not their long term priority or strategy.
Its imperative we move away ASAP
Llama.cpp (library which ollama uses under the hoods) has its own server, and it is fully compatible with open-webui.
I moved away from ollama in favor of llama-server a couple of months ago and never missed anything, since I'm still using the same UI.
totally respect your choice, and it's a great project too. Of course as a maintainer of Ollama, my preference is to win you over with Ollama. If it doesn't meet your needs, it's okay. We are more energized than ever to keep improving Ollama. Hopefully one day we will win you back.
Ollama does not use llama.cpp anymore; we do still keep it and occasionally update it to remain compatible for older models for when we used it. The team is great, we just have features we want to build, and want to implement the models directly in Ollama. (We do use GGML and ask partners to help it. This is a project that also powers llama.cpp and is maintained by that same team)
I’ve never seen a PR on ggml from Ollama folks though. Could you mention one contribution you did?
> Ollama does not use llama.cpp anymore;
> We do use GGML
Sorry, but this is kind of hiding the ball. You don't use llama.cpp, you just ... use their core library that implements all the difficult bits, and carry a patchset on top of it?
Why do you have to start with the first statement at all? "we use the core library from llama.cpp/ggml and implement what we think is a better interface and UX. we hope you like it and find it useful."
6 replies →
> Ollama does not use llama.cpp anymore
That is interesting, did Ollama develop its own proprietary inference engine or did you move to something else?
Any specific reason why you moved away from llama.cpp?
1 reply →
So I’m using turbo and just want to provide some feedback. I can’t figure out how to connect raycast and project goose to ollama turbo. The software that calls it essentially looks for the models via ollama but cannot find the turbo ones and the documentation is not clear yet. Just my two cents, the inference is very quick and I’m happy with the speed but not quite usable yet.
2 replies →
Fully compatible is a stretch, it's important we dont fall into a celebrity "my guy is perfect" trap. They implement a few endpoints.
They implement more openai-compatible endpoints than ollama at least
I won't use `ollama` on principle. I use `llama-cli` and `llama-server` if I'm not linking `ggml`/`gguf` directly. It's like, two extra commands to use the one by the genius that wrote it and not the one that the guys just jacked it.
The models are on HuggingFace and downloading them is `uvx huggingface-cli`, the `GGUF` quants were `TheBloke` (with a grant from pmarca IIRC) for ages and now everyone does them (`unsloth` does a bunch of them).
Maybe I've got it twisted, but it seems to be that the people who actually do `ggml` aren't happy about it, and I've got their back on this.
It’s unfortunate that llama.cpp’s code is a mess. It’s impossible to make any meaningful contributions to it.
I'm the first to admit I'm not a heavy C++ user, so I'm not a great judge of the quality looking at the code itself ... but ggml-org has 400 contributors on ggml, 1200 on llama.cpp and has kept pace with ~all major innovations in transformers over the last year and change. Clearly some people can and do make meaningful contributions.
Interesting, admittedly, I am slowly getting to the point, where ollama's defaults get a little restrictive. If the setup is not too onerous, I would not mind trying. Where did you start?
Download llama-server from llama.cpp Github and install it some PATH directory. AFAIK they don't have an automated installer, so that can be intimidating to some people
Assuming you have llama-server installed, you can download + run a hugging face model with something like
And access http://localhost:8080
Isn't the open-webui maintainer heavily against MCP support and tool calling?
hmm, how so? Ollama is open and the pricing is completely optional for users who want additional GPUs.
Is it bad to fairly charge money for selling GPUs that cost us money too, and use that money to grow the core open-source project?
At one point, it just has to be reasonable. I'd like to believe by having a conscientious, we can create something great.
First, I must say I appreciate you taking the time to be engaged on this thread and responding to so many of us.
What I'm referring to is a broader pattern that I (and several) others have been seeing. Of the top of my head: not crediting llama.cpp previously, still not crediting llama.cpp now and saying you are using your own inference engine when you are still using ggml and the core of what Georgi made, most importantly why even create your own version - is it not better for the community to just contribute to llama.cpp?, making your own propreitary model storage platform disallowing using weights with other local engines requiring people to duplicate downloads and more.
I dont know how to regard these other than being largely motivated out of self interest.
I think what Jeff and you have built have been enormously helpful to us - Ollama is how I got started running models locally and have enjoyed using it for years now. For that, I think you guys should be paid millions. But what I fear is going to happen is you guys will go the way of the current dogma of capturing users (at least in mindshare) and then continually squeezing more. I would love to be wrong, but I am not going to stick around to find out as its risk I cannot take.
Everyone just wants to solarpunk this up.
In an ideal world yes - as we should - especially for us Californian/Bay Area people, that's literally our spirit animal. But I understand that is idle dreaming. What I believe certainly is within reach is a state that is much better than what we are in.
2 replies →
I believe that is what https://github.com/containers/ramalama set out to do.
Huggingface also offers a cloud product, but that doesn’t take away from downloading weights and running them locally.
Oh no this is a positively diabolical development, offering...hosting services tailored to a specific use case at a reasonable price ...
They can’t keep getting away with this.
Yes, better to get free sh*t unsustainably. By the way, you're free to create an open source alternative and pour your time into that so we can all benefit. But when you don't — remember I called it!
What? The obvious move is to never have switched to Ollama and just use Llama.cpp directly, which I've been doing for years. Llama.cpp was created first, is the foundation for this product, and is actually open source.
But there's much less that works with that. OpenWebUI for example.
3 replies →
> Its imperative we move away ASAP
Why? If the tool works then use it. They’re not forcing you to use the cloud.
There are many, many FOSS apps that use Ollama as a dependency. If Ollama rugs, then all those projects suffer.
Its a tale we seen played out many times. Redis is the most recent example.
Most apps that integrate with ollama that I've seen just have an OpenAI compatible API parameter which defaults to port 11434 which ollama uses, but can be changed easily. Is there a way to integrate ollama more deeply?
1 reply →
Local inference is becoming completely commoditized imo. These days even docker has a local models you can launch with a single click (or command).
i was trying to remove it but noticed they've hidden the uninstall away. It amounts to doing a rm - which is a joke.
happy sglang user here :)
I stopped using them when they started doing the weird model naming bullshit stuck with lmstudio since