Comment by trollbridge
11 hours ago
This is already an issue I face both with clients and personally where I can say something authoritatively, but they’ll always go and just ask an LLM.
Sometimes it’s in my favour. LLMs estimate very high prices and long timelines for software projects.
If it’s a legal / regulatory issue it’s usually wrong.
It's still mildly amusing when I ask what kind of work is involved for a change, and it gives me some estimate of 3-6 weeks or whatever, and then I say "no I mean like what changes" and then it spits out the complete change in 30 seconds.
That has been my experience as well. One-shot time and budget estimates are always ridiculously off.
If you hook it up to legal databases like Lexus Nexus and let it ONLY search proper places for law (instead of Wikipedia) these systems are superhuman at law too. A lawyer acting as a meat proxy for Opus 5.5 will smoke a lawyer without AI systems on average these days.
Yes, well, a typical client has a free tier of ChatGPT and just voice-to-text’s questions into that.
Was anyone doubting LLms are good at searching and summarising text?
Some internal engineer is doing god's work by creating a system skill for the LLM that multiplies all estimations by five. /s
On a more serious note, the pre-LLM internet had very little useful detailed information regarding software estimation addressing effort for building something in specific domains.
Most writing on software estimation tried to tackle a general approach that could apply to all domains. So you had a "applicable to all, useful to none" situation going. And since crud apps were the most ubiquitous apps, software estimation techniques were perceived to give more accurate results with respect to those type of apps.
I'm sure internally big AI companies are accumulating vast amounts of estimation training data by documenting how long it takes their LLM to complete specific app building tasks. I don't think they would be in a rush to push that data into public models though as it would be like beating a dead horse that is the current human workforce anyway.