Comment by parliament32
17 hours ago
Well of course it'll eventually work, just a random text generator will eventually produce code that passes your tests if you run it hard enough.
The problem is it's devouring your tokens as it does so. While you're on a subsidized plan that seems like a non-issue, but once the providers start charging you actual costs for usage.. yeah, the hallucinations will be a showstopper for you.
If you can point me to any "random text generator" that scores a 76.8 on SWEBench, after any number of iterations, or in fact is competitive on any benchmark at all, I'll happily switch to it. Until then, I don't think that analogy will lead to particularly productive conversation. There are many engineers using a similar harness on LLMs today. No one uses a random text generator to generate code because you are not making a real suggestion.
> The problem is it's devouring your tokens as it does so. While you're on a subsidized plan that seems like a non-issue, but once the providers start charging you actual costs for usage.. yeah, the hallucinations will be a showstopper for you.
The discussion, and your original post, was about whether hallucinations are a meaningful issue today - not in some hypothetical future.