Comment by holden_nelson

I got the same result when I tried the "remove the walls" example. I followed up by telling it to reimplement up/down/left/right movement without reinserting the walls and it basically made the square with the flag "unenterable". Like with a force field.

Respectfully I'm not sure if I share your worry though. You would have the same problem if you wrote extremely thorough / exhaustive unit tests or extremely precise types and didn't allow the LLM to amend them. You're basically shrinking the state space of what is considered a "correct" program per your spec. That the LLM has to get extremely creative to break your program is a _feature_; in my experience, an LLM does not have to get creative at all to break a typical program written in an enterprise setting, and that's unfortunate.