Comment by meltyness
6 hours ago
This is a weakness of docker, a bit, I think.
I was rigging this up, myself, and conciscious of the fact that basic docker is "all or none" for container port forwarding because it's for presenting network services, had to dig around with iptables so it'd be similar to binding on localhost.
The use case https://github.com/meltyness/tax-pal
The ollama container is fairly easy to deploy, and supports GPU inference through container toolkit. I'd imagine many of these are docker containers.
e: i stand corrected, apparently -p of `docker run` can have a binding interface stipulated
e2: https://docs.docker.com/engine/containers/run/#exposed-ports which is not in some docs
e3: but it's in the man page ofc
Yes the binding interface can be specified, but the default for -p 11434:11434 is 0.0.0.0.
IMO the default should be 127.0.0.1 and the user should have to explicitly bind to all via -p 0.0.0.0:11434:11434.
Apparently been that way for a while haha
https://github.com/moby/moby/commit/1cbdaebaa1c2326e57945333...
Docker has a lot of lazy hacks to make it work well on MacOS that had to have it running in a VM for any of the linux containers to work.
Out of curiosity, why would you need to wrap the call to an Ollama modelfile in docker? Does the dockerized ollama client provide some benefit, when it’s shelling down to local Ollama instance anyway? (Wrt tax-pal)
It's more of a distribution thing for me really. I'm basically using docker as a package manager since they otherwise distribute through one of those ad-hoc shell scripts that I'd prefer to avoid accidentally breaking Debian with somehow.
I've built ollama before too, but, I like that I can cleanly rip it out of my system or upgrade it without handing root off to some shell script somewhere I guess.
If anyone's gonna bash up my system it oughta be me