Comment by radarsat1

wav2vec or similar approaches are used a lot these days, which is basically BERT with audio inputs. Whether that counts as an LLM or not, I am not sure. It's a transformer architecture in any case.

People will often reach for "easily trainable, cheap" solutions when they can; the reason people reach for Transformers and LLMs is because when you throw more data at them, they get better.