Comment by Tanjreeve
12 hours ago
That bug is fixable and verifiable. The LLM you cross your fingers till the next time the same thing happens.
12 hours ago
That bug is fixable and verifiable. The LLM you cross your fingers till the next time the same thing happens.
> fixable and verifiable
By people with a specific skill set. LLMs generation can also be fixed and verified by people with a certain skill set, and non-deterministic computing doesn't automatically mean unpredictable. When people say that the LLMs are a black box, it means unpredictability in unknown situations.
You do structured output, input validation, output validation, lower temperature, limit decisions, RL, etc. to increase predictability to near certainty. It's just statistics after all. Or you can as well generate the code to do the job.
It's just that the required skill set is a different one to do those things, and unusual in the context of DB administration.
None of the things you mention are guaranteed to increase the probability of correctness. You can run the LLM output through as many deterministic programs as you like, but "the query plan runs in acceptable time" is not something you can verify with such a tool. Nobody knows how the LLM does it, so they cannot know how to make the LLM do it better.
> Nobody knows how the LLM does it
From a completely technical perspective, we have a rough idea how the LLMs work, and improving a system requires measuring outcomes and you don't necessarily need to understand the mechanism.
EXPLAIN ANALYZE against data that's similar in size to prod checks a query written by an LLM as good as anything we can write... but, yes, you're right, we still didn't solve the halting problem - neither the LLMs.
Even if the query plan was not generated by an llm, you can't verify it will run in an acceptable time. This is one of the biggest unsolved problems in databases
Only if someone is planning on running a pinned self hosted version of an LLM alongside the DB to fix the problem. The developer can change the binary easy enough and test it but the LLM approach just seems either theoretical or bending ourselves in knots to justify using an LLM.
I didn't argue that it'd make sense, I just said that it could be reasonably fixable when problems occur and verifyable that the fix works. Even if shipping and RLing an LLM were easy tasks in terms of software distribution (they are not) we'd still hit the skill mismatch, as I said in my previous comment.
I just find the "all llms are non dererministic and therefore unreliable" narrative a bit backwards. All software that has more than 0 users needs to deal with non-determinism anyway :)
> That bug is fixable and verifiable
A bad query plan is not your typical kind of bug. I would definitely not call it fixable. Query planners are inherently dealing with estimations and approximations. If the query planners estimation is off, you're screwed.
Unless you come up with a way to cheaply determine exactly how many rows a query will return, bad query plans will still exist.
I've reduced queries from minutes long to 2s in Postgres by duplicating a CTE and keeping it unused
Query planner feels pretty LLM-esque already