Comment by fivetenpen
4 hours ago
The biggest issue with business users (who don’t know SQL) writing SQL with an LLM is that there is no one to validate that query and now that business user will treat that LLM response as canon to share in meetings, presentations, and with clients. The LLM may have forgotten a filter, used the wrong definition of revenue, or misunderstood the user’s intent by writing a query that answers their question in the most literal way.
That is the crux of the issue. No amount of semantic layers and context will help this until an LLM can read the user’s mind to remove ambiguity in the prompt.
I see most of the benefits of LLMs to be used by analysts who know SQL to work more productivly.
also, the same metric can be used in different ways and for different purposes across a business, so you might need different validation rules and definitions.
Maybe one separate semantics validation layer could help, but costs 2x or possibly Nx if you need to recover and turn a wrong query into a correct one