Comment by simonw

> Human managers are inherently gullible; we've got no plausible path to unbias them. LLMs have at least one plausible path which is to train them to be a little bit cynical.

Firm disagree on claims 2 and 3 (paths to unbias each), though I agree humans (managers included) are inherently gullible.

There's a lot of research into human biases and how to overcome (or at least mitigate) those biases; and one can in principle always hire a "no man" to look for things which can go wrong. This is kinda what corporate lawyers (and, I hear, corporate economists) are there for.

AI, unfortunately, have a weakness which isn't present in meat-based intelligence, one which won't go away even if we get brain-uploads to copy meat-minds into silicon to make better AI: the very fact of being cheap enables us to find their weaknesses by spamming a bajillion variations at them to see what slips past their cognitive blind-spots.

Unfortunately, my take on the second paragraph is even more cynical than yours:

> We're not going to call it "management" necessarily, but there is no question that LLMs are going to take over decision making from managers eventually. Why choose a monkey guessing what the evidence says you should do when you could have an optimised evidence-weighted statistical model making the bets? The only reason to use humans is there are still technical limits on how general the models are, limits that seem to be falling away at a pleasing rate.

We're already seeing LLMs take over decision making from managers, not because they're good in the "optimised evidence-weighted statistical model" sense, but because they're good in the "hyper-persuasive to lazy primate brain" sense.

This also shows the limits of the "hire a no-man" strategy, as this is happening despite the list of people saying "aaaaa this is dangerous!" including many of the people developing these particular AI models, along with some Nobel laureates, various campaign groups marching around with placards, and a bestselling book.

Interesting that humans can't be trained to improve ("be less biased") but AI can. I would say this is a much more damning conclusion for the AI replacing ICs than managers.

Whats easier, training AI to make good bets (what does that mean in business? Make the most money? Worker quality of life? World a better place?) or training it to get code to compile?

If we can just simulate the business accurately enough we can solve having to interact with the market… which is also trying to solve interacting with us… We just need to do it more accurately…

Something tells me people will still be in the mix here.

> "optimised evidence-weighted statistical model"

Isn't such a model inevitably going to be lagging what is happening?

(Monkeys can see/smell/recognise the scat or track of a large cat very quickly and don't sit around to check the data)