Comment by saberience
16 hours ago
What you're talking about is not synthetic data. Synthetic data is data generated by an LLM. If the data is generated by a human, it's not synthetic.
You're talking about real data created and curated by humans to help in training LLMs.
This is true and thank you for correcting me. I make a mistake of using synthetic data for what is human-generated data. There are augmentation tricks they could use on the human data but still human-generated data shouldn't be mistaken with synthetic data. I believe my core arguments still stand. Specifically that these models are learning their capabilities through specialized data that human professionals curate. There is no emergence or "general" intelligence. This is really what the open-weight models lack.