Comment by transcriptase
21 hours ago
I use advanced voice a lot and have come across many weird bugs.
1) Every response would be normal except end with a “whoosh” like one of those sound effects some mail clients use when an message is sent, and the model itself either couldn’t or wouldn’t acknowledge it.
2) The same except with someone knocking on a door. Like someone would play on a soundboard.
3) The entire history in the conversation disappearing after several minutes of back and forth, leading to the model having no idea what I’m talking about and acting as if it’s a fresh conversation.
4) Advanced voice mode stuttering because it hears its own voice and thinks it’s me interrupting (on a brand new iPhone 16 Pro, medium-low built in speaker volume and built-in mic).
5) Really weird changes in pronunciation or randomly saying certain words high-pitched, or suddenly using a weird accent.
And all of this was prior to these most recent changes.
It also stutters and repeats sometimes and says poor connection even though I know the connection is near-ideal.
I may know why that first one happens! They’re not correctly padding the latent in their decoder (by default torch pads with zeros, they should pad with whatever their latent’s representation of silence is). You can hear the same effect in songs generated with our music model: https://sonauto.ai/
Yeah we’re too lazy to fix it too
I’m super curious now, how does padding lead to repeatedly ending tts replies with what seem to be an actual non-speech sound effect?
If you pad your output with something that doesn't represent silence, then any outputs that happen to have a non-standard length (i.e. nearly all outputs) will end with whatever sound your padding bits represent in the model's embedding space. if "0000" represents "Whoosh," then most of your outputs will end in "whoosh."
Here's a non-AI example: If all HN comments had to be some multiple of 50 characters long and comments were padded with the letter "A," then most HN comments would look like the user was screaming at the end. AAAAAAAAAAAAAAAAAA
1 reply →
In addition to what Centigonal said, even if the autoencoder was trained on only speech data, an all zero vector is probably just out of distribution (decoder has never seen it before) and causes weird sounds. However, given the hallucinations we're seeing, the AE has (maybe unintentionally) likely seen a bunch of non-speech data like music and sound effects too.
> 4) Advanced voice mode stuttering because it hears its own voice and thinks it’s me interrupting
I experience the same issue on an iPhone 15 Pro Max and have to mute the mic whenever I'm listening to a response. I wish they added an option to disable voice interruptions so that it could be interrupted only by touch.
Oh man, I thought the “whoosh” was an intentional indicator that it was done speaking.