Comment by tw04
5 hours ago
>It's kinda cool to see a whole lot of otherwise intelligent people who are so dogmatically and ideologically opposed to anything AI that they're going to willfully dismiss anything that AI produces regardless of utility.
You'd probably put me into that bucket, although I'd disagree. I'm not at all against using AI to do something like: type up a high level summary of a product featureset for an executive that doesn't require deep technical accuracy.
What I AM against is: "summarize these million datapoints and into an output I can consume".
Why? Because the number of times I've already witnessed in the last year: someone using AI to build out their QBR deck or financial forecast, only to find out the AI completely hallucinated the numbers - makes my brain break. If I can't trust it to build an accurate graph of hard numbers without literally double checking all of its work, why would I bother in the first place?
In the same way, if you tell me you've got this amazing dataset that AI has built for you, my first thought is: I trust that about as much as the Iraqi Information Minister, because I've seen first hand the garbage output from supposedly the best AI platforms in the world.
*And to be clear: I absolutely think businesses across the board are replacing people with AI, and they can do so. And I also think it'll take 18+ months for someone to start asking questions only for them to figure out they've been directing the future of their company on garbage numbers that don't reflect reality.
Asking an LLM to analyze data directly doesn’t work. But they’re great at writing scripts to analyze (and visualize) data. Anthropic just figured this out last week and gave Claude a mode that does that for you.
This. I only ask LLMs to summarize non-critical stuff, i.e. just give me a general summary of all the work done over the past week.
If I were in need of hard analytics you can be damn sure I'd have it build a tool with a solid suite of tests following a rigorous process to ensure the outputs are sound. That's the difference between engineering and vibing.
Yes, you have to calibrate the effort to the task, you can't just blindly vibecode it. But if you treat it like a new college hire who still remembers their stats course, rather than a senior analyst who will just come back with the right answer, you can do some pretty high-level stuff that's trustworthy. It's so fast that it's no problem to double/triple check everything and even do it with multiple methods.