Comment by aithrowawaycomm

Reading the paper, this seems like putting the cart before the horse: the agents individually are not actually capable of playing Minecraft and cannot successfully perform the tasks they've assigned or volunteered for, so in some sense the authors are having dogs wear human clothes and declaring it's a human-like civilization. Further, crucial things are essentially hard-coded: what types of societies are available and (I believe) the names of the roles. I am not exactly sure what the social organization is supposed to imply: the strongest claim you could make is that the agent framework could work for video game NPCs because the agents stick to their roles and factions. The claim that agents "can use legal structures" strikes me as especially specious, since "use the legal structure" is hard-wired into the various agents' behavior. Trying to extend all this to actual human society seems ridiculous, and it does not help that the authors blithely ignore sociology and anthropology.

There are some other highly specious claims:

- I said "I believe" the names of the roles are hard-coded, but unless I missed something the information is unacceptably vague. I don't see anything in the agent prompts that would make them create new roles, or assign themselves to roles at all. Again I might be missing something, but the more I read the more confused I get.

- claiming that the agents formed long-term social relationships over the course of 12 Minecraft days, but that's only four real hours and the agents experience real time: the length of a Minecraft day is immaterial! I think "form long-term social relationships" and "use legal structures" aren't merely immodest, they're dishonest.

- the meme / religious transmission stuff totally ignores training data contamination with GPT-4. The summarized meme clearly indicates awareness of the real-world Pastafarian meme, so it is simply wrong to conclude that this meme is being "transmitted," when it is far more likely that it was evoked in an agent that already knew the meme. Why not run this experiment with a truly novel fake religion? Some of the meme examples do seem novel, like "oak log crafting syndrome," but others like "meditation circle" or "vintage fashion and retro projects" have nothing to do with Minecraft and are almost certainly GPT-4 hallucinations.

In general using GPT-4 for this seems like a terrible mistake (if you are interested in doing honest research).