Comment by etherealG

moshi by Kyutai seems to have beaten your approach by about 500ms, and they're going to release open source.

https://news.ycombinator.com/item?id=40866569