Comment by kqr
5 hours ago
This surprised me too. I suspect the robots only got a "use mutation testing" prompt and independently decided that it must mean manual mutation testing, or possibly that they ran in a sandbox where an automated mutation testing tool was not installed.
But I'm surprised Dan Luu didn't make a remark about this. Surely he must know the difference!
He did remark on it: "Although mutation testing is a standard programming term, agents generally didn't actually do mutation testing and instead did normal testing with some small amount of mutating things in a way that isn't really mutation testing, similar to how the TDD instruction modified behavior but didn't get agents to do TDD."
If you're wondering why he doesn't go into greater detail, several testing methodologies earlier we have: "Since just saying that agents didn't really meaningfully do the thing is repetitive, I'll make these sections short and only highlight particular curiosities."