Comment by ofjcihen
15 days ago
I agree with most of what you wrote except for this:
>Frequent LLM users already know not to do that.
And I think that’s the biggest problem. Amidst the current push to utilize LLMs across orgs and groups there are a large (if even say majority) of people that are using them every day but who have never approached anything as technical as a “harness” before let alone an entire setup.
For them the behavior mentioned here is a major issue.
Exactly. When I use a scissor, I don't want the scissor to not work just because I'm not a "frequent scissor user," and then get told by someone who makes their breakfast with scissors that I'm doing it wrong. Most people will not be "frequent" anything users.
Most people also understand that, because they're not "frequent" users of a thing, they absolutely suck at using it, and set their expectations accordingly. In particular, they realize that doing anything non-trivial with the thing requires them to spend some learning and practice time, or asking/hiring a "frequent" user to do it for them.
So the reasonable response to being told you're holding your scissors wrong is to realize that yes, you most likely are holding your scissors wrong[0], and ask the other person for advice (or just to do the thing), or look up a YouTube video and learn, or sign up to a class, or such.
Expecting mastery in 30 seconds is not a reasonable attitude, but it's unfortunately the lie that software industry tried to sell to people for the past 15 years or so.
--
[0] - There's much more to it than one would think.
I’m interested in the “non-trivial” point as well, this seems to be a common refrain from the anti-LLM tech crowd, “LLMs aren’t good at doing anything non-trivial”, well is that really the case or is it just harder and one needs to put in more practice for more complicated tasks?
I don’t have an example off hand, but I know that it’s easy to dismiss something an LLM does as trivial if your work is extremely marginal. Most devs aren’t creating their own programming languages. I can’t help but think people who hold this opinion also think the work most software professionals do is “trivial” (“you’re just moving strings around, that’s not impressive/trivial”)
2 replies →
If you make the example any more complicated, it makes sense.
A lathe operator isn’t any good if they don’t frequently operate lathes.
A articulated robot implementer needs frequent experience implementing robots to be any good.
That doesn’t mean lathes or robots are useless. Nor does it mean they have failed as products because they require expertise.
I do think it raises questions as to whether vast swathes of the population will be effective at using LLMs. Are they scissors, or a lathe?
I figure english is the next coolest programming language for scripting and compilation. So far people have been writing fun little demos with it, but now people are starting to place real demands on it, and you're starting to see actual programs needing to be built. Unsurprisingly this requires a bit more craft.
14 replies →
Everybody seems to want them to be scissors, or at least to treat them as such, but even still the reason everyone can use scissors so well is because they’ve practiced with them, right? You’re probably a lot better at using scissors now than the first time you did it, the functionality is just so simple it’s harder to notice.
To me learning to use LLMs is the same as doing anything else, you have to practice and put in the hours to get good. Maybe some harnesses will eventually allow LLMs to function more as scissors than lathes. This seems to be what Microsoft is trying to do by embedding Copilot in all their products and saying “choose the UI that works best for you”. If that doesn’t end up working we’ll need another paradigm for “non-technical” users to effectively operate computer assistants
Scissors tend to be reasonably straightforward, so there your analogy seems to hold, but upgrading to chopsticks, needles, and chainsaws tends to attract increasing amounts of "you're doing it wrong" in increasingly alarmed tones of voice.
Exactly - I am a lawyer and we are told to use dedicated AI products as much and however we want. There will be errors made
Much to the often-reported chagrin of judges across the country.