Comment by jclarkcom

It's a good point and I should make a distinction on what models are appropriate. I think of chatGPT 4 like a college student and chatGPT 5.1 5 Pro (deep thinking model) more like a seasoned professional. I wouldn't trust non-frontier, non-thinking models with a result for this kind of question. But the determinism of the result does not scare me, the out output may vary but not directionally. The same thing would happen if you asked the foremost security expert in the world, you'd get slightly different answers on different days. One time as a I test I ran a very complex legal analysis through chat GPT pro 10 times to see how the results would vary and it was pretty consistent with ~10% variation in numbers it suggested.