Comment by brettdav
5 hours ago
Can you share a bit more about your bifrost and llama swap setup? I’m facing memory constraints and am looking for a managed model solution that will help with hot swapping loaded models and stay-warm concurrency. Ideally with prioritization.
What do you want to know? Just start llama-swap with the models i have downloaded, add llama-swap as a provider in bifrost, expose the models you want and they become available in one single endpoint you can use in anything like opencode, openwebui or anything that speaks openai.