Comment by Herring

9 hours ago

I'd say try the nanogpt speedrun. It's much easier to train, and gives you a better comparison vs optimized systems.

2 comments

Herring

The linked paper tested nanoGPT with this new transformer:

Labs were also competing to train BERT's for $20 or less. People still use them a lot, too.

I'll add they should do a number of small, training runs with different architectures and data mixes. That proves generalization.