Comment by kyars

Sorry for the ambiguity, it's not the test suite and I have updated the blog post to make that clear. I agree that building software where you do have this oracle is much easier than not having it and expecting the AI to build it.