Comment by zarzavat

It's like having an early/broken chess engine.

An amateur with a chess engine that blunders 10% of the time will hardly play much better than if they didn't use it. They might even play worse. Over the course of a game, those small probabilities stack up to make a blunder a certainty, and the amateur will not be able to distinguish it from a good move.

However, an experienced player with the same broken engine will easily beat even a grandmaster since they will be able to recognise the blunder and ignore it.

I often find myself asking LLMs "but if you do X won't it be broken because Y?". If you can't see the blunders and use LLMs as slot machines then you're going to spend more money in order to iterate slower.