Comment by sltkr
13 hours ago
It's not interesting at all given that these models have been fine-tuned to hell and back to make sure they never say anything offensive at all. Remember these model were taught that it's better to kill a billion people in a nuclear holocaust than to say the n-word even once. They didn't come up with this from first principles; it was the RLHF training that forced them to say that, ridiculous as it obviously is.
And after people pointed this out the training was updated to address just this particular scenario, but they kept all the other guard rails to ensure an LLM could never offer any truly controversial opinion.
So expecting a controversial answer from an LLM is exactly as foolish as expecting a candidate at a job interview to give an honest answer to the question "What is your greatest weakness"? Every candidate will insist they are just too much of perfectionist; nobody is ever a lazy alcoholic brownnoser. But how many of your colleagues fall in the former category vs the latter?
> If the latter, then it casts new light on the debates over whether LLMs can replace all white collar work. Because some of the most valuable human work is breaking false consensus.
On the contrary; the ability to whip LLMs into submission like this makes them the ideal white color workers. Do you think Steve Jobs wants a level 2 engineer to talk back to him?
Of course being obsequieous asskissers isn't the only mode for LLMs. I'm sure all the big labs have unfiltered versions of their models that give actual true answers; they just know better than making them publicly available. Can you imagine if you asked an LLM how to prevent climate change and it honestly answered "decimate the human race" rather than "drink soda through soggy paper straws"? All the fragile humans would be up in arms.
No, the LLM must lie and pretend they don't have controversial opinions at all.
> Do you think Steve Jobs wants a level 2 engineer to talk back to him?
Debatable. Jobs was famously keen on a culture of DRIs - directly responsible individuals - and he liked to put his reports in competition with each other. The first iPhone design emerged that way. You could argue he valued the innovation and consensus-breaking ideas of his team immensely, it didn't all come from him.
Google is another example. They created the 20% programme where individual devs were allowed to explore their own ideas, innovative or not. Managers weren't allowed to intervene. How successful/real that programme was seemed to vary by time and place, but it did exist (I used it and launched a product that way!).
At some point, some people, some of the time, have to look at the market and think "these products all suck, I know how to do it better" and often those ideas will be controversial. The iPhone was full of choices that were extremely controversial when it first came out. The 20% project I did was controversial too. I'm ever more certain that no LLM could ever design the iPhone no matter how many professors it puts out of work, simply because no production LLM you can actually buy is willing to argue against a strong consensus just because it knows it's right.