Comment by awesome_dude
15 days ago
AI uses a high confidence tone - likely because its training data is heavy on authoritative texts/reference books.
And it does get people into a lot of trouble.
I have got into trouble with it when it is extremely confident about something I am not very familiar with (as recently as two weeks ago with Claude). I have also had long drawn out "arguments" when I have known it's wrong based on my experience and intuition, and it has steadfastly refused to take my point (last week)
I have learnt to ask it why it was doing something that has turned out to be incorrect, as a post-mortem, and it's all apologetic and subservient and "never going to do that again" (but still does as soon as the context window shifts [eg. run git commands, or, yesterday, kept telling me to use commands that were explicitly communicated to Claude as not being available, and completely wrong - I was shifting from one tech stack to another and Claude kept telling me the original commands, not the new ones])
I'm expecting Claude to be a better search engine - I have spent literal years (if not decades) knowing that asking the right question is what's required to get the right answer, and LLM's natural language processing is what's supposed to make that easier than using Google or grep, or even Stack Overflow - but the reality is that I still have to be on my toes, especially when I am drifting into territory I am unfamiliar with.
>And it does get people into a lot of trouble.
Pretty much everyone takes it at face value unless we know otherwise from prior experience. Even the most advanced models make embarrassing mistakes and fumble with simple tasks. Yet we are very willing to give them exceptional slack for it? I wish I knew why. Are people just that easily overcome by confident voices?
> Are people just that easily overcome by confident voices?
Back in high school, my AP calculus class did some experiments with our teacher's blessing. We'd send a kid out to walk around during class and see how long it took for them to get sent back. Anyway, it ends up that walking around purposely with a piece of paper or envelope, like you're on a mission to deliver it, was a very successful tactic.
I've seen internet comments similar: "put on a yellow vest and carry a clipboard and you can enter any building, anywhere." Confidence is scary, and often misleading.
3 replies →
I find it really disturbing, I think because it is illuminating a much more basic problem. It is there in our political and religious histories. We're living through a similar political time right now. A large number of people seem all to ready to find some pervasive authority and subjugate themselves to it.
The more concrete machine authority figure is also prevalent in scifi literature. Sometimes, I am not even certain if the author is doing this to examine this issue versus just leaning into it as either appealing to themselves or to the perceived audience.
Conversely - we tell people who are speaking in public to "Show confidence" - or in job interviews "Hire people who are confident"
We've also pushed back "The more a person knows, the less confident they are" - Dunning Kruger - often used to dismiss over confident people - points out that people are really confident, at first, then that confidence drops away, markedly, but it rebuilds (slowly).
That last rise in confidence is what (I believe) people use as a heuristic on the likely level of knowledge possessed by the speaker (AI or human)
Most engineers know, though, that overconfident people are toxic - the difference between arrogance and genuine confidence in the answer is incredibly difficult to define.
Yeah - I don't know /why/ but, as I say, I've been guilty of that myself, very recently, despite knowing it's a shockingly poor guide when left to its own devices.
Maybe because when it's right it actually expands my knowledge - there have been genuine instances where it's gone - something to the effect of - "Yo, there's this other idea for approaching the problem" which has turned out to be exactly what I was looking for?
At least for me, the answer is that despite the mistakes and the sheer annoyance the prose causes me, they are unbelievably useful. I accomplished multiple major achievements in the last two years that most probably wouldn't be possible at all, surely not within that timeframe.
> I have also had long drawn out "arguments" when I have known it's wrong based on my experience and intuition, and it has steadfastly refused to take my point (last week)
Ironically, trying to argue with Claude about the limitations of LLMs and AI in general today is quite hard. It refuses to yield, likely due to Anthropic tweaking it aggressively