Comment by WorldMaker

The trouble is that "just parse the tests" isn't always an option and running arbitrary code is the nature of how software is built.

The easiest example is JS testing. Most test harnesses use a JS file for configuration. If you don't know how the harness is configured how do you know you are parsing the right tests?

Most test frameworks in JS use the define/it `define("some test colection", () => it("some test", () => /* …test code… */))` pattern. Tests are built as callbacks to functions.

In theory, sure, you could "just" try to RegEx out the `define("name"` and `it("name"` patterns, but it becomes harder to track nesting than you think it is with just RegEx. Then you realize that because those are code callbacks, no one is stopped from building meta-test suites with things like `for (thing of someTextMatrix) { it(`handles ${thing}`, () => /* …parametric test on thing… */ }`.

The test language used most in JS is JS. It's a lot harder problem than "just parsing" to figure out. In most cases a test harness needs to run the JS files to collect the full information about the test suite. Being JS files they are Turing Complete and open to doing whatever they want. Many times the test harnesses are running in a full Node environment with access to the entire filesystem and more.

Most of that applies to other test harnesses in other languages as well. To get the full suite of possible tests you need to be able to build that language and run it. How much of a sandbox that language has in that case shifts, but often it is still a sandbox with ways to escape. (We've proven that there's an escape Zero Day in the Universal Turing Machine, escapes are in some ways inevitable in any and all Turing Complete languages.)