Comment by CaseFlatline
7 hours ago
I am trying to find how the synthetic data was created (looking through the repo) and didn't find it. Maybe I am missing it - Would love to see the prompts and process on that aspect of the training data generation!
It's here:
https://github.com/arman-bd/guppylm/blob/main/guppylm/genera...
Uses a sort of mad-libs templatized style to generate all the permutations.