Comment by xscott
3 days ago
Probably not what you're after, but I've considered having a separate small mm-model act as a seeing-eye dog for the bigger more capable one.
3 days ago
Probably not what you're after, but I've considered having a separate small mm-model act as a seeing-eye dog for the bigger more capable one.
But how do you prompt this smaller model to give back information? I've tried that in the past but didn't go well. What I do is to send written handoff files between models to pass context around, but only had good results with big vision models as well.
So many possibilities for how you could glue it all together. However, when I send Gemma 4 12B in llama.cpp an image with no accompanying text, it assumes I want a description and gives me one.
I just tried with an audio file, and it transcribed the lyrics as I hoped. Then it made a bunch of suggestions about what do next, which I wasn't after. I could probably fix that by sending some text to narrow the scope.
I've tried this, too, and for whatever reason, simple things like having a subagent with vision capabilities hallucinates responses to the blind models. I don't know why, and it's not consistent, but I don't see this happening when the primary agent is a vision model.