Comment by Aeolun

A lot of the mentioned tools are fuzzers[1], these are tools that start from certain input(s), say "SELECT * FROM TABLE WHERE some_column = some_value", and repeatedly

1- mutates[2] the input,

2- measures some metric of the database's performance (e.g. how many non-syntax errors it spat out)

3- and introduces more or less mutations to increase or decrease that metric.

In short, an evolutionary computation where the population being generated is test code, the fitness function is some metric summarizing how the generated test code tested the db code, and mutation operators gradually pushes the generated test code towards a certain optima that we want our tests to have.

This is an extremely general approach that can be used to generate anything, and it's uncannily effective at finding bugs especially in structured-format consumers like compilers[3] and db engines. You can use it to generate queries as above and the fitness function would be something like "Did the database gave an unholy error squeak or violate some key invariants in the db file?", you can use it to generate general test harnesses that exercise the whole C code and measure coverage via instrumentation, the fitness function would be something like "How much did this mutation covered of the application's instructions?". Not only do they take both of those approaches and many others, but they do it multuple times and independently, so for example it's mentioned that Google and the core development team both have an independent query fuzzer. They mention about 4 or 5 fuzzers doing similar thing.

And off course fuzzing is just one way to generate tests, there are plenty others.

>I strongly doubt it’s all actually source code

I mean, why not? source code is not necessarily hand-written source code.

[1]: skim https://www.fuzzingbook.org/ for a fairly good overview

[2]: Generally, mutation is either completely blind and general, i.e. byte-level , or structure-aware, i.e. has some notion of a grammar or a schema governing the thing it mutates, it wouldn't just change a random letter of "SELECT", because the result would almost cerainly be invalid SQL.

[3]: https://embed.cs.utah.edu/csmith/

Databases are extremely complex.