Comment by cryptonector
1 year ago
Oh no, SQLite3 is a lot more featureful than SQLite2. The proprietary test suite is what makes SQLite3 so solid.
1 year ago
Oh no, SQLite3 is a lot more featureful than SQLite2. The proprietary test suite is what makes SQLite3 so solid.
This makes me wonder. Is anyone practicing TDD with genAI/LLMs? If the true value is in the tests, might as well write those and have the AI slop be the codebase itself. TDD is often criticized for being slow. I'd seriously like to compare one vs the other today. I've also heard people find it challenging to get it to write good tests.
I'd sort of invert that and say it's better to use LLMs to just generate tons more test cases for the SQL DBs. Theoretically we could use LLMs to create 100s of Thousands (unlimited really) of test cases for any SQL system, where you could pretty much certify the entire SQL capability. Maybe such a standardized test suite already exists, but it was probably written by humans.
At that point, you'd get a ton more value from doing Property Testing (+ get up and running faster, with less costs).
If I'd had to have either code or tests generated by a LLM, I'd manually write the test cases with a well-thought out API for whatever I test, then have the LLM write tests that implements what I thought up, rather than the opposite which sounds like a slow and painful death.
1 reply →
Are you sure that LLMs, because of their probabilistic nature, would not bias against certain edge cases. Sure, LLMs can be used to great effect to write many tests for normal usage patterns, which is valuable for sure. But I'd still prefer my edge cases handled by humans where possible.
1 reply →
TDD suffers from being inflexible when you don't fully understand the problem. Which on software is basically always.
Everytime I've tried it for something I make no progress at all compared to just banginf out the shape that works and then writing tests to interrogate my own design.
Happy that it's not just me. I tried it a couple of times, and for small problems, I could make it work, albeit with refactorings both to the code and tests.
But for more complicated topics, I never fully grasped all the details before writing code, so my tests missed aspects and I had to refactor both code and tests.
I kinda like the idea more than the reality of TDD.
4 replies →
dude, if you have the llm write the tests, then you have no confidence it's testing what you think it is. Making the test worthless
Dude, I was suggesting that you, not the LLM, write the tests in this scenario.