Comment by karaterobot

This is a student thesis, which doesn't mean it's bad, but is worth keeping in mind. I see that their control group size was three people (in three different sessions), but I don't see how many people they ran through the AI part of the experiment.

I was wondering the other day whether there were good services for running tabletop games over AI. I don't know about good, because I haven't tried them, but there are certainly a few products like this today. Given that ChatGPT plus some fine-tuning was comparable to a human GM in this experiment, I'd be interested in another experiment that looked at whether these products (which charge more money) offer any benefit over just using ChatGPT. For them to be better than ChatGPT, it seems like they'd almost have to be better than a human GM to be worth the money.