Comment by noodletheworld

Alas, I was desperately keen to read transcripts of actual gameplay and all we got was user surveys?

Was there a structured combat system as sort of this or was it a talking only kind of thing? Who knows?

Trying to get empirical evidence that an LLM can even remotely plausibly work as a DM with no social aspect and (as far as I can tell, reading the paper) no programmed system of stats, rolls, combat etc. seems impressive.

…but was it that? Or just a DnD themed chatGPT conversation? Or a text adventure with an LLM describing each “room”?

Did the LLM actually have agency to make decisions and perform tasks, or was it not an agent at all, just a conversation?

Without a transcript, all you can really say for sure is yeah, I agree; more research is needed.