Comment by yhjc2692

13 days ago

Wow this is really impressive. Throwing away the VAE feels counterintuitive - I'm shocked it works so well! Appreciate that you shared your training insights on the blog post!

Reminds me of this great paper where they trained a LLM that beats LLaMA2-13B from scratch with < $0.1M: https://arxiv.org/abs/2404.07413

Does it? Text generation models have been decoder-only for a while, with great benefits to their training. It's surprising to me that more attempts weren't made for other modalities.