Comment by gpjt

I'm adding MoE support to the GPT-2 code from Sebastian Raschka's "Build a Large Language Model (From Scratch)". Just the minimal changes. Planning to do (or at least start!) a from-scratch base model training run on my own hardware before the end of the month.