Comment by torginus

I think the reference to scaling is a pretty big giveaway that things are not as they seem - I think it's pretty clear that we've run out of (human produced) data, so there's nowhere to scale to in that dimension. I'm pretty sure modern models are trained in some novel ways that engineers have to come up with.

It's quite likely they train on CC output too.

Yeah, there's synthethic data as well, but how do you generate said data is very likely a good question and one that many people have lost a lot of sleep over.