Comment by ben_w

> Human managers are inherently gullible; we've got no plausible path to unbias them. LLMs have at least one plausible path which is to train them to be a little bit cynical.

Firm disagree on claims 2 and 3 (paths to unbias each), though I agree humans (managers included) are inherently gullible.

There's a lot of research into human biases and how to overcome (or at least mitigate) those biases; and one can in principle always hire a "no man" to look for things which can go wrong. This is kinda what corporate lawyers (and, I hear, corporate economists) are there for.

AI, unfortunately, have a weakness which isn't present in meat-based intelligence, one which won't go away even if we get brain-uploads to copy meat-minds into silicon to make better AI: the very fact of being cheap enables us to find their weaknesses by spamming a bajillion variations at them to see what slips past their cognitive blind-spots.

Unfortunately, my take on the second paragraph is even more cynical than yours:

> We're not going to call it "management" necessarily, but there is no question that LLMs are going to take over decision making from managers eventually. Why choose a monkey guessing what the evidence says you should do when you could have an optimised evidence-weighted statistical model making the bets? The only reason to use humans is there are still technical limits on how general the models are, limits that seem to be falling away at a pleasing rate.

We're already seeing LLMs take over decision making from managers, not because they're good in the "optimised evidence-weighted statistical model" sense, but because they're good in the "hyper-persuasive to lazy primate brain" sense.

This also shows the limits of the "hire a no-man" strategy, as this is happening despite the list of people saying "aaaaa this is dangerous!" including many of the people developing these particular AI models, along with some Nobel laureates, various campaign groups marching around with placards, and a bestselling book.