Comment by zamadatix
7 hours ago
I'm taking 0.90^n and 0.93^n and seeing 0.93 does more than the initial 3% better over time, which preciesly saves you from things like spending tokens on garbage by this larger factor. You're free to pick whatever target reliabilities from that. E.g. you can maintain above 80% accuracy over 3 turns rather than 2 or 50% over 6 turns instead of 3 or you can pick another percentage point and see how the depth to expected error/expected error at depth numbers are much better than 3% and how another 3% actually doubles accuracy because this is all non-linear.
There's also the matter of "90% compared to what". If it was 90% per attempt compared to a calculator then I'd just use a calculator. If it was 90% per attempt for something I have a 10% chance of getting right per attempt then it's not really so bad. OTOH if it's 90% per attempt at keeping my car in the lane then it's obviously quite bad. I.e. there isn't one "reliable percentage" to target which is why I'm comparing the curve instead of just the most stringent things I can think of in isolation.
Completely agree that error tolerance depends on what it is that we're doing. Some things demand no errors at all, some things have a cost to attempt, some things you can't pre-evaluate for correctness if you don't already have the prerequisite knowledge and you only get one chance. Not everything is software where we can conjure 50 more end-to-end tests to ward off demons. When it's not a harmless throwaway weekend project, 7% is already a huge error rate before you ever think of feeding it back into itself.