Comment by CodesInChaos
17 hours ago
Since the library claims to be postgres compatible, I wonder if its developers ran the existing postgres jepsen suite against it (possibly with some adjustments). Or why not.
Especially now, where you could probably just ask AI to adapt the existing suites to the cosnistency claims this database makes, and get something useful out of it. While certainly not as good as a professional test, I'd expect it to at least find some low hanging fruits before going for the expensive and public official Jepsen test.
The company says they did. From the post
"Orkes said they ran Jepsen’s existing Postgres test harness to find and resolve issues in ParticleDB prior to this work. However, that same test harness failed immediately when run against ParticleDB 1.1.4. Orkes also stated they tested ParticleDB with SQLancer, which generates far more complex queries than the ones we tested. It is therefore perplexing that nearly every feature we tested exhibited glaring errors, and that our findings seemed to come as a surprise."
They probably asked an AI to run the tests against their DB. And the AI said "yeah I ran it, all green!" (except with a lot more words and perfect grammar, and maybe a few section headings and emoji.)
Seems to me that “we ran the Jepsen test” is just something an LLM thought would sound good on a DB’s website.
Assuming the LLMs did actually run the tests, I think this points to the flaw in TDD and LLMs following goals blindly.
Tests are usually not comprehensive. The tests for SQL engines for example aren't intended to provide full code coverage, because they are intended to be re-usable across different codebases. LLMs can therefore easily over-fit to those tests, solving problems in the dumbest ways rather than solving them in the ways humans would.
The human's goal is to produce a working database engine, the LLM's goal is to pass the test suite or implement the spec it has been given.