Comment by padolsey
2 years ago
Hmm well I believe reddit made up a huge portion of the training data for GPT2 but yes, tbh I have no support for the claim that that's the case with current versions. Anyway, I guess if we consider a forum as following the general scaffold of human conversation, it's a good analogy. But yes there's a tonne of other content at play. If we consider, "where does chatgpt inherit its conversational approach from?" .. that may be a good approach. Almost nowhere in human prose, from either journals or novels, is there an exchange where a tip is seen as inviting a more verbose or detailed conversational response. It's kinda nonsensical to assume it would work.
The conversational approach is deliberate via fine tuning and alignment.