Comment by Lucasoato
8 hours ago
I’ve tested at least 20 STT models in a benchmark I’ve set up with German, Italian and English voices from meetings in my company.
The voices contained very industry specific words, the languages changed from one sentence to another, sometimes words in a language were mentioned while a discussion was in another.
The only local model that satisfies me is Voxtral Mini 3b, the only paid API that is slightly better is eleven labs. Yes, Voxtral might not reach the best score in the benchmarks, but to me, it just solves a problem. It might not be the best in terms of speed... but that’s not a problem for me.
Happy to test this new model from Google but I’m not sure I’d go with that instead of something that can run so easily in my machine.
This problem is legitimately hard and needs high cognitive abilities. Even the biggest generalist models struggle with memes and lingo salad that sound immediately intuitive for an out-of-the-loop human, and I'm talking about text comprehension. Modern models are optimized for decision making and are worse in that than old ones optimized for creative writing, but those also struggled.
You should probably not expect a small STT model like Voxtral or Parakeet to do any better, unless it's laser focused on that area in particular and sucks in everything else.
Agree about Voxtral being the only model — local or cloud — that handles multilingual conversation really well. I‘m not sure what they do differently, but mixed-language sentences and industry terms don‘t seem to faze it where other model begin to struggle.
Do you use it on a desktop? Mac by any chance? What's your setup?
I've been looking to find a simple and fast dictation app for English but almost everything I've tried (from Handy to many apps, eg some with Whisper in their names, after the model I assume) just don't work well. Apple's offering is worse than those though. I even tried with local enhancement models.
I am biased as a developer on the project but you should give Epilude [1] a go if you want local dictation on a Mac.
We’ve fine-tuned open-weight models to make them better (in our benchmarks) at cleaning up and formatting what you say so you don’t have to edit what you dictate.
https://epilude.com
You should give VoiceInk[1] a go. It works great and has decent latency.
https://github.com/Beingpax/VoiceInk
have you tried groq whisper-turbo? works well for me for multiple different languages
I use Parakeet 3. How does that fare in your testing?
I love it but it misses the business specific words when in different language. Sometimes it stretches them out to fit unrelated words in the language of the rest of the conversation. I miss its speed though.
Any possibility to share some of the eval audio?
Of course not, but it’s easily replicable just by mixing different languages conversations together, adding a word here and there of some very specific German jargon.
Try the recent MOSS one? It’s very good