Comment by Lerc
23 days ago
I'm glad someone did this, but It doesn't surprise me that the models are not up to particularly advanced story writing yet. I could see them writing some rather dry high-concept sci-fi fairly soon, but there are some types of story with interpersonal relationships that I cannot imagine a non sentient writer being able to produce. I wouldn't rule it out an AI being capable in the future, but I think if it did, it would mean we would have to seriously consider that it's a sign of a solid understanding of emotions and theory of mind.
I found LLMs very good at reading human chat logs and doing psychological analysis of the participants, and of the interplay
hard to imagine how they do that without a good understanding of emotions/theory of mind
if they pattern match to psychology books, that is still functional understanding since they can operationalize them
They definitely have some. They handle the basic test scenarios where multiple people alter some state without the knowlege of the other. Most models seem to handle tracking differing knowledge between entities.
Some of the simple bench tests show how thin that can be.
Things like two lifelong friends had a fight when they were 8 years old because A broke B's favourite toy. They are now 25 and B sees A drowning, Will B try and save A?
Models tend to place massively undue influence on facts just because they have been mentioned. If every part of the text is accurate and relevant, then this helps immensely. Many fine tuning examples are precisely on topic. That leads to a bias against ignoring trivia.
Fine tuning has gotten a lot better now that the value of nuance in training examples is better known.