Comment by vntok
6 days ago
Open WebUI works perfectly fine with llama.cpp though.
They have very detailed quick start docs on it: https://docs.openwebui.com/getting-started/quick-start/start...
6 days ago
Open WebUI works perfectly fine with llama.cpp though.
They have very detailed quick start docs on it: https://docs.openwebui.com/getting-started/quick-start/start...
Oh thanks I didn't know that :O
I do also need an API server though. The one built into OpenWebUI is no good because it always reloads the model if you use it first from the web console and then run an API call using the same model (like literally the same model from the workspace). Very weird but I avoid it for that reason.
llama.cpp is what you want. It offers both a web UI and an API on the same port. I use llama.cpp's webui with gpt-oss-20b, and I also leverage it as an OpenAI-compatible server with gptel for Emacs. Very good product.