Comment by leftouterjoins
14 hours ago
This is my experience. As a data engineer with years in my current role I can get AI to write good SQL that correctly answers questions or defines metrics. At worst it may take a few back and forths to really nail it.
However, if a PM tries it they will more often than not get a wrong answer because they do not know what to look for or how to ask the right questions.
Prompts still matter a lot especially when you are working in an established enterprise.
What model are you using? I find Claude Sonnet to be quite capable of generating correct queries. Also, are you using a semantic layer [1], because without it the model may have trouble understanding your schema. I would be very surprised if you PM gets a wrong answer with these settings. I have never seen Sonnet 4.6+ produce a bad query.
[1] https://communitysupport.group/t/what-is-the-purpose-of-the-...