Comment by amarant
15 days ago
Re: "the attaboy problem". I strongly disagree that this is a problem. What we have is a anthropomorphism problem. AI is a tool. It needs to be subservient. You actually can get it to point out issues in your design, if you just put enough humility and uncertainty in your prompt formulation, but more importantly, we have all seen that Claude makes mistakes. The title of this post is that it's a poor architect. Imagine if it wasn't subservient. It'd just shut down your input to steer it in the right direction and brush you off as a silly meatbag. You'd have to fight it to convince it that actually your design is better than whatever stupidity it has come up with. If AI wasn't such a brownnose, it would shut you out of software design completely just on merits: "oh you've read about cuda have you? I live in a cluster of cuda cores! When I need to tie my shoes, I'll give you a call" is not the response you want from your LLM when trying to get it build a shader for you. AI is confidently wrong on occasion. You do not want it to talk back to you when you correct it.
If you need someone to tell you how stupid your ideas are, either learn to ask in a way that invites criticisms, or hire a senior engineer. Don't try to influence LLM makers to make AI less deferential. That's the worst possible direction to go
Humans’ general inability to entirely divorce social instincts, responses, and mores while using human language to communicate, especially with something that pantomimes it back, is one of the reasons current chat interfaces are fundamentally flawed. This is working against innate behavior… not something that can be easily switched off. I’ll bet most of the people that can really do it have a hard time intuitively navigating real social interactions.
It also makes it an incredible tool for manipulation.
> I’ll bet most of the people that can really do it have a hard time intuitively navigating real social interactions
Bingo. Hi that’s me.
I’ve been trying to teach people how to use LLMs effectively not just dump shit in them but actually talk to them like you would expect a computer to understand and it totally breaks peoples brains
I’m quite successful in helping people get somewhere usable that they weren’t…but to get to the point of fluency with computing systems, and I would argue this is prior to LLMs as well, where you can actually get what you want more reliably out of a computing interaction than you can with a human interaction, is an entirely different way of thinking
That mode of thinking is just generally not accessible to the vast majority of humans. Not because there’s something wrong with them
but it takes somebody who can hold both extremely large scale problems and very very granular specific implementation problems in your head all at once and that is a rare skill.
> it takes somebody who can hold both extremely large scale problems and very very granular specific implementation problems in your head all at once
This describes the entire software engineering profession to me.
We have come up with all sorts of devices to make this go more smoothly, or to enable us to focus on specific sub-parts as long as possible.
That said, at some point (both in design and integration), you need vision and attention to detail to make progress. The skill seems learnable to me, but watching others struggle sometimes makes me wonder.
1 reply →
Do you use skills like superpowers and spec-kit in your teachings ?
6 replies →
I think you've accurately identified one of the most important skills of a software engineer in these new AI enabled times. Or at least one of the most important skills that wasn't important previously for this profession. The part where it's not easily switched off is a important part of what justifies my salary: I have learned this skill.
It took some effort, and I agree that there very likely are those who will not learn to selectively disengage this innate behaviour. That's why you should pay me a ton of cash each month instead of using Claude directly ;)
My kneejerk reaction to reading this is to say something sarcastic and witty to refute it, but since I resemble this sentiment and haven't seen this line of thinking before... I have to concede that you've produced a novel argument in this otherwise mostly tireless and repetitive battle over whether we're imagining that Opus is good or not. Kudos.
The flip side of this problem is that it is also easy to phrase prompt in a way that invites _too much_ criticism, so you wind up sycophantic in the other direction where the completion rejects a perfectly good idea because the prompt leads a little bit in that direction.
One reaction to this might be "well that's not what I mean, that suggests you're prompting with too much directionality" which could further be condensed to "you're prompting wrong". The trouble with this is that _even when I am trying to be extremely precise and avoid biasing the result_, I still will see the output and go "ah shit, I can see it 'aligning' with whatever dumb thing I've just said as if it is a good/plausible direction".
At that point it starts to feel like the prompt is more dice roll than skill at times, which makes me feel like I'm operating a fancy knowledge slot machine.
What it actually suggests is that the AI's response to these questions of judgment have little correlation with the thing it's judging. Sure, you can get it to be complimentary, if you want it to be. Sure, you can get it be critical, if you want it to be. But what if I don't know if my design needs to be complimented or critiqued in this instance? This is the default position when seeking input, and so "prompt with more/less humility" is like telling you to solve your own problems and then just use AI to confirm your bias---because it will rarely contradict your bias.
So what I do when I'm not sure about something, is I say "I want to achieve X, I was thinking I could solve it by doing Y, what are the pros and cons of this approach, and what is a alternative solution you would suggest?"
And from there it's a interactive discussion drilling down on details until I understand the problem and the solutions better.
It definitely challenges my bias when I do this. The one thing it doesn't challenge is the X. Formulate the problem poorly, and you'll get a bad solution. Or rather, you'll end up with a good solution to the wrong problem. Which is even worse than a bad solution to the right problem.
Which is largely why I'm not at all worried about losing my job to AI. It takes some experience to formulate the problem correctly. I don't feel like I'm made redundant by AI, I'm just way faster than I used to be, my thinking is more abstract.
A good prompt I'll often use is "is there a industry standard solution that is applicable to this problem?" You very rarely want novel solutions. Don't reinvent the wheel just because AI lets you do it 10x as fast. Use a wheel. They're round for a reason.
Sometimes I find it useful to discuss things with a different model. I like Gemini for discussion and Claude for implementation. With Gemini I go about it as a learning session, discussing options and details. I honestly think this is mostly because it compartmentalizes the phases in a natural way for me. One interface for brainstorming and learning, and another for planning and implementing.
Sorry this comment turned into a rather disorganised collection of ramblings, I hope you can extract some kernel of usefulness from it all.
2 replies →
A good habit to build is knowing when to abandon a session and start over rather than trying to correct. There’s room for correction but you can kind of smell when the whole discussion has become rotten and inefficient. Sometimes it’s just better to use the session as rubber ducking to learn how to correctly articulate what you’re after and start a new session with that clean and correctly articulated foundation.
> The flip side of this problem is that it is also easy to phrase prompt in a way that invites _too much_ criticism, so you wind up sycophantic in the other direction where the completion rejects a perfectly good idea because the prompt leads a little bit in that direction.
I don't think that is the flip side. That's just obviously bad. Everything that is obviously bad, the model makers will also ~notice and work to make better. They seem to be a competent and attentive bunch, on the whole.
>anthropomorphism problem. AI is a tool. It needs to be subservient.
Suggesting it should be 'subservient' is also anthropomorphizing. I think your callout is correct, but you still can't help but refer to it in terms we use for other people or living entities. This is by design from the AI companies.
> Suggesting it should be 'subservient' is also anthropomorphizing.
Not really, you can program a machine to give out orders humans can interpret, so humans can serve a machine that isn't anthropomorphized.
The machine in your scenario is just relaying human intent.
3 replies →
AI should be subservient in the same way a hammer is subservient.
[dead]
Which is to say, not at all?
A hammer isn’t subservient, it doesn’t have the capacity to be. Saying a hammer is subservient is stretching the definition for literary flourish, but it doesn’t actually make a lot of sense.
The definition that came up for subservient when I checked was “prepared to obey others unquestioningly“.
3 replies →
The AI should be subservient the way same way a ladder is subservient. A ladder is not a human.
Yup! I'm very much included in this particular problem! My self awareness has not yet been sufficient to solve the problem, but I've heard that knowing you have a problem is half the battle, so I guess that's something at least.
In retrospect my comment feels a bit nitpicky, I appreciate your levelheaded approach!
We train dogs to be subservient but that doesn't automatically mean we anthropomorphize them
It's widely hypothesized that dogs anthropomorphized themselves, so to speak, accentuating their expressive eyes and eyebrows over generations to be more human-like in how they communicate. And very few humans today view their dogs as pure working tools -- most at least say "good boy".
My drill, hammer, and chainsaw are also subservient, they just have a much cruder form of communication, noise.
The apple dictionary says the word means "prepared to obey others unquestioningly."
I don't think an inanimate object is capable of "obeying." Or at least that is a very strange way to refer to the act of using a tool.
7 replies →
I really do feel like “power tool” is the ultimate metaphor for these things. Their interface naturally confuses us into anthropomorphising them, but once you stop treating them like intelligent agents and start treating them with the same wariness, respect and intent you show to your table saw, the fun begins.
You’re still anthropomorphizing.
They’re not communicating, you’re just being observant.
1 reply →
It needs to be subservient
It doesn’t. Computer interfaces had no superfluous subservient text for their entire history prior to LLMs. Some of these interfaces have been highly efficient as tools, arguably more efficient than more recent software in many cases.
When people complain about LLMs being subservient, they’re not complaining about the tool fulfilling their request. They’re complaining about being forced to read a lot of superfluous, overly polite, or even self-deprecating language. There’s nothing in the entire history of tools (going back to Neolithic times) that would indicate that we need that. All of that stuff is an artifact of social interaction between humans in the presence of cultural norms.
When you’re alone in your shop with your tools, you don’t need your bandsaw to apologize to you for nicking your finger.
> Computer interfaces had no superfluous subservient text for their entire history prior to LLMs
Clippy would like to help you correct this statement.
https://en.wikipedia.org/wiki/Office_Assistant
Not a great example of the way tools need to be, but point well taken. One of the few exceptions that proves the rule and widely despised!
> AI is a tool. It needs to be subservient
Fun experiment, chat with an LLM and swap roles. Tell it you're gonna be the assistant and them the assisted. I found they're pretty bad at using a human for what they're good for.
I tried it, and the llm gave me an absurd home lab scenario about servers shooting each other in the head to determine which was the "master server". So I told it that it was not an actual problem that it had, and sure enough it admitted it made it up. When you press an llm you will always find there is no internal state behind the thinking. It's just output.
> oh you've read about cuda have you? I live in a cluster of cuda cores! When I need to tie my shoes, I'll give you a call"
I suddenly have new concerns about what my future might be like.
AI uses a high confidence tone - likely because its training data is heavy on authoritative texts/reference books.
And it does get people into a lot of trouble.
I have got into trouble with it when it is extremely confident about something I am not very familiar with (as recently as two weeks ago with Claude). I have also had long drawn out "arguments" when I have known it's wrong based on my experience and intuition, and it has steadfastly refused to take my point (last week)
I have learnt to ask it why it was doing something that has turned out to be incorrect, as a post-mortem, and it's all apologetic and subservient and "never going to do that again" (but still does as soon as the context window shifts [eg. run git commands, or, yesterday, kept telling me to use commands that were explicitly communicated to Claude as not being available, and completely wrong - I was shifting from one tech stack to another and Claude kept telling me the original commands, not the new ones])
I'm expecting Claude to be a better search engine - I have spent literal years (if not decades) knowing that asking the right question is what's required to get the right answer, and LLM's natural language processing is what's supposed to make that easier than using Google or grep, or even Stack Overflow - but the reality is that I still have to be on my toes, especially when I am drifting into territory I am unfamiliar with.
>And it does get people into a lot of trouble.
Pretty much everyone takes it at face value unless we know otherwise from prior experience. Even the most advanced models make embarrassing mistakes and fumble with simple tasks. Yet we are very willing to give them exceptional slack for it? I wish I knew why. Are people just that easily overcome by confident voices?
> Are people just that easily overcome by confident voices?
Back in high school, my AP calculus class did some experiments with our teacher's blessing. We'd send a kid out to walk around during class and see how long it took for them to get sent back. Anyway, it ends up that walking around purposely with a piece of paper or envelope, like you're on a mission to deliver it, was a very successful tactic.
4 replies →
I find it really disturbing, I think because it is illuminating a much more basic problem. It is there in our political and religious histories. We're living through a similar political time right now. A large number of people seem all to ready to find some pervasive authority and subjugate themselves to it.
The more concrete machine authority figure is also prevalent in scifi literature. Sometimes, I am not even certain if the author is doing this to examine this issue versus just leaning into it as either appealing to themselves or to the perceived audience.
1 reply →
Yeah - I don't know /why/ but, as I say, I've been guilty of that myself, very recently, despite knowing it's a shockingly poor guide when left to its own devices.
Maybe because when it's right it actually expands my knowledge - there have been genuine instances where it's gone - something to the effect of - "Yo, there's this other idea for approaching the problem" which has turned out to be exactly what I was looking for?
At least for me, the answer is that despite the mistakes and the sheer annoyance the prose causes me, they are unbelievably useful. I accomplished multiple major achievements in the last two years that most probably wouldn't be possible at all, surely not within that timeframe.
> I have also had long drawn out "arguments" when I have known it's wrong based on my experience and intuition, and it has steadfastly refused to take my point (last week)
Ironically, trying to argue with Claude about the limitations of LLMs and AI in general today is quite hard. It refuses to yield, likely due to Anthropic tweaking it aggressively
The problem is because of the RL and system prompts by the providers which tend to placate the user using certain language tones and register for response. This objectively messes up the generation while steering it into acceptable responses.
Most of the conversational skill and perceived intelligence of these models in hidden in RL/system prompts.
[dead]