Comment by hamilyon2
15 hours ago
Optimal plan construction is math-heavy, algorithm-heavy and vary even by workload. There are options like creating just-in-time indexes, so solution space grows even faster than article presents. Sometimes it is the query planner which is the slow part of total execution time.
LLM is kind of blunt weapon to use here. I am waiting rather for alphago style neural net heuristic.
I also wondered why an LLM would be the right starting point. Why would Balzac or billions of lines of rwir code or reddit be relevant to mapping this smallish, well-defined language (SQL) to this other tiny constrained specification language (the query plan suggestions)? You could make a (relatively) tiny network and then actually pass it some relevant features of the actual data, like as numbers, not just as text returned from a tool call.
It’s ultimately based on a lot of hand-written heuristics. Google has some non-LLM based machine learning technique to guide optimization heuristics in LLVM; that would be closer to what you are looking for.
I think your CPU might even have a small neural net in the branch predictor
What if we use a hybrid model of using both query optimizer and LLM? Whichever produces better result, the database can use?
- a question from someone with lack of DB depth, me.
This is about to bake your noodle:
https://www.postgresql.org/docs/current/geqo-pg-intro.html
GEQO is not to get a better plan than the traditional optimizer, it is to be able to get a plan at all when the query is large. And it's widely known for creating poor plans.
2 replies →
The immediate problem: How do you know which one is better without running them?
You create formulas to estimate the cost of running a given query plan. Use statistics collected about the tables (e.g. how many rows) to try to be accurate. The topic is "Cost Based Optimization".
3 replies →
Could you A/B at random, use that to collect data and eventually feed that back in to prefer A or B depending on the shape of the query?
3 replies →
[flagged]
3 replies →