Comment by sixtram
4 days ago
You say you can have increasingly nuanced discussions with stronger models.
What I say is, when I asked Claude why he applied a certain change I didn't understand, and boy, it was a small change, he said he "reasoned from first principles" based on the code paths. But it didn't work, and when I asked, "Okay, describe the steps of your reasoning from first principles," it literally answered that it had just made it up.
So, nuanced discussions with models, I don't buy it.
You can never ask why a model did a certain thing, or what it was "thinking" when it said something - just like you can't ask a human which neurons were firing when they had a certain thought. The information just isn't available at that level.
You absolutely can have deep nuanced discussions with LLMs however, you just need to better understand their strengths and weaknesses.
A human won't respond with "Neuron 10-100 of the frontal cortex" (jokes aside) with deceptively convincing confidence.
The human will quite convincingly be able to construct a post-hoc reasoning on an action that may or may not be related at all to what was actually going through their head or the actual instinctual reasons that led to a decision.
16 replies →
You certainly can ask it what it was thinking, the problem is just that it's more likely to make up a plausible sounding fabrication than to say "I don't know" or "my reasoning is hidden for business reasons" (frontier models hide a lot of their chain of thought). Which is the fundamental problem with LLMs though, if the data doesn't exist or it's sparse they make things up.
Choosing plausible sounding fabrication over an admission of ignorance is not an uncommon modality among the human beings I interact with, so I'm not surprised this pattern is found in models trained on human interactions.
1 reply →
So you're saying I can absolutely have a deep, nuanced discussion with an LLM, as long as I don't ask how he arrived at his conclusions?
You can also have a deep nuanced discussion with a rubber duck as long as you don't ask any questions it needs to respond to.
Have you not seen all the posts with claims that AI lies about its reasoning when asked to explain how it arrived at the output?
I would instead ask the model to explain how X works, whether it achieves Y, and why we cannot do Z instead.
That is how you have a discussion with the AI.
You can have a nuanced discussion with an LLM. But LLMs also have failure modes where they start making up justifications. The two are not mutually exclusive.
>as long as I don't ask how he arrived at his conclusions?
So just the average US political discussion with a human then?
> You can never ask why a model did a certain thing
Of course you can! It might be following outdated docs or read something in legacy code and tried to follow that pattern and it'll tell you as much if you ask it in a way that actually gets you the reason instead of it thinking it needs to immediately fix the mistake.
Dude, these two things are not at all analogous:
1. Asking a model why it did a certain thing, and
2. Expecting a human to say which neuron fired in their response.
Even asking a human being why they did a certain thing is questionable. The research on choice blindness seems like a pretty definitive debunking of post-hoc rationalization:
https://en.wikipedia.org/wiki/Introspection_illusion#Choice_...
2 replies →
"Nuanced discussion" doesn't necessarily mean the sort one would have with a human. Statistical apologies are never going to be meaningful. One could edit nonsense into the context window and the model would attempt to rationalize it. The models are smart but you need to use them in a way that makes sense for what they are.
"Nuanced discussions" is more about describing a design to a model, asking the model to critique your design and ask you for clarifications, and then you providing those clarifications and the model "getting it" and proceeding to additional levels of detail before implementation. In particular the models being able to highlight concerns you have not yet thought about is a pretty good sign of this. Fable is noticeably better at this compared to Opus.
I was not talking about models making mistakes. Mistakes, and then models making up justifications for those mistakes, is a failure mode of any LLM, and Fable is no different in that regard. Newer models might make less mistakes, or at least make less egregious mistakes, but they still make mistakes.
Posts like this are meaningless without more context - the model you're using, the harness, the initial prompt and context.
Fable is better than most staff engineers at my FAANG.
Maybe I’m missing something, but he talks about charm and tasks (repos on his GitHub). Charm being his harness, and tasks being one of his skills. Idk, maybe I’m mistaken from reading the article…
https://github.com/taoeffect
> Fable is better than most staff engineers at my FAANG.
While this wouldn’t entirely surprise me, my experience is just not that. Using Claude and fable, it regularly (poorly) recreates features that exist inside our codebase. Sure, I could give way more initial context but at a certain point I’ve given so much context that I would have been faster writing the code myself, or I could have literally handed it to even a fresh graduate to write.
> Fable is better than most staff engineers at my FAANG.
That’s genuinely disturbing.
But staff engineers take "responsibility"
Including you?
Fable will definitely be the one on call when it inevitably breaks down from the pile of shit slop it wrote at 5AM, don't worry <3
We already use AI for oncall and it works better than our humans most of the time.
> he
:/
[flagged]
We can point out mistakes that feel rather grating without assuming intent behind them.
I agree that their use of "he" is likely because they're not a native speaker, especially because they're arguing against the capabilities of LLMs.
That doesn't make it inherently wrong to point out the mistake when it's so intertwined with the deeper discussion here, especially given the fact that some (hopefully few) people do build relationships with LLMs.
> turd bucket autist
I’d be more willing to engage with your argument in good faith without inflammatory language like this. Try and meet people where they are and these conversations become easier.
5 replies →
That may be true but it’s still capable of nuanced discussions.