Comment by chopete3
11 hours ago
This is so right. We training Whisper Large model on 20,000 audio samples specific to a domain and it ended up reducing the ASR by 5% while improving WER of the finetuned domain by 0.5%.
Instead we ended up with no finetuning. We give audio snippet to 2 AsR models, take 3 best transcriptions and ask the LLm to pick the best based on the context. That produced significantly higher accuracy in how an agent understands the users.
Can you go into more detail on this? I’ve been putting together the data pipeline for an ASR fine tune but your approach sounds more interesting.
basically he is feeding the same input to multiple models, taking their outputs and dumping it into an LLM to sort out what the actual transcription probably is. expensive but effective.
Deep Fusion is best, when words and phrase patterns in the domain are known. Deep Fusion means to hint the Whisper decoder about the next possible words using LLM-in-the-loop.