Comment by danielmarkbruce

The comment below/above is right. Pre-training isn't the path (but you really do need to learn how it's done). Learn how it works, learn the architecture, then take models and do post training. Fine tuning and RL on them. There was a fun/clever/useless project recently posted here about building a terminal emulating model thing, as an example.