Comment by svat

It's a great question. But note that Google Translate is also trained on "predict the missing token": https://blog.research.google/2022/05/24-new-languages-google... / https://arxiv.org/abs/2205.03983 (search the blog post around “Surprisingly, this simple procedure produces high quality zero-shot translations.”)

This was in May 2022, as part of Google Translate adding support for several low-resource languages (including Sanskrit). I was already very surprised that simply training on predicting tokens does translation so well — then a few months later ChatGPT came out, trained (roughly) the same way and doing a lot of things besides translation.