Comment by conradfr
13 days ago
Call me crazy but sometimes I still find a better solution on StackOverflow than what Claude Code insists to do.
I'm not sure we're better off without SO in the long run.
13 days ago
Call me crazy but sometimes I still find a better solution on StackOverflow than what Claude Code insists to do.
I'm not sure we're better off without SO in the long run.
Same here. LLMs are great at spitting out well-known solutions to problems instead of the best one. The "long tail" of solutions is usually lost due to how tokens are sampled from the LLM's probability distribution.
What I found to help a lot is to ask for e.g. 10 different solutions to a problem and then choosing one of them. Sometimes, this even leads to borderline creative solutions if there aren't 10 different ones.
In theory reasoning tokens should do the equivalent of this - explicitly create options outside of the quick-response probability space, so those can guide future generation.
In practice, models that do this won't be prioritized as much, because the economics of thinking tokens that stop by default at, say, one option plus a bit more planning (short of full alternatives) would be superior as long as billing is per-user instead of per-token. So we'll still need to play games with prompting!
Without continuous feedback from real world, lower-probability token (and soon high-probability ones as well) will be complete garbage.
> LLMs are great at spitting out well-known solutions to problems instead of the best one.
I remember how Stack Overflow would close questions as duplicates just because somebody suggested the wrong answer that is also the right answer to the existing question. The best way to get a correct answer on Stack Overflow (and forums before that) was to post the wrong answer as part of your question.
One thing that SO had was you could see multiple solutions and implementations for something. Sometimes the "best" solution isn't very readable code, sometimes you are able to understand the problem better when you see a bunch of people solving it in different ways and arguing about it like angry monkeys.
It really could be bad though.
The bot can do that kind of thing, too.
"Show me 6 very different solutions, and present arguments for/against each one as if a bunch of angry monkeys."
SO has always had a pretty strong stance against opinion-based questions, but this is maybe the niche they should be exploring now. Humans still have a lot to say about the "best" solution to a given problem. The whole idea of an "accepted" answer could be removed, for example, since that's what AI will already generate.
Much of what Claude insists you do probably came from SO.
Or Reddit. I don't know about Claude but Gemini has given me answers that are verbatim comments from Reddit.
I've gotten my own answers given back to me for problems I forgot I already had.
2 replies →
Claude does it quite a bit when you’re triggering the search tool functions.
It’s fine, and what you would expect for certain prompts, except that the synthesized results often come back communicating more authority than they deserve.
1 reply →
I've gotten this too a lot. If you ask AI to cite where it got info you can lose a lot of confidence in it pretty quickly.
I have seen it quote my own code back at me, including comments word for word.
Yeah, that's basically it.
In robotics there is no free lunch dataset. You'll have to gather it yourself, but if you do that, you run into an obvious problem: labeling.
With SO, you literally have the best possible scenario, because the data is clearly structured and separated into prompt and answer (aka label).
Your objective function is literally "Say what he said".
I've seen chatgpt word for word plagiarise stack overflow answers
i've seen it plagiarize personal blog posts too, almost verbatim code line by line.... kind of shocking...
1 reply →
I'm not sure we're better off without SO in the long run.
You're right but that site has been sputtering culturally for some time. I put a lot of effort into editing questions and answers on ServerFault (part of SO) but I feel that time was wasted. I think they knew for a while they just wanted to sell it and just stopped caring. A number of editors were allowed to be jerks for too long and it went to their heads. I wish I could take back all that effort.
well, SO is probably the highest quality data source for a language model and the rest of the internet is just diluting the final latent space limited by Jon Skeet.
What you are noticing in a long term is the "community" knowledge and communication which the chatGPT is now kind of destroying. In some sense, it is no different from the difference between studying along and studying with your peers at a university.
You can definitely study alone and achieve perfect grades, but studying with your peers is how you build relationships for future life and take your community forward as a whole.
was just gonna post the same thing
needed to implement a language feature that was a bit complicated and im not familiar with it so just planned with claude to do it, and after each write/fix cycle it just wouldn't work right.... gave up, went back to SO copy pasted the (not perfect but enough to start from) answer and worked up from there...
at the same time my knowledge grew and im more confident to do this same capability myself whereas reiterating with claude it was just a slog and i didn't learn much...
i think i may be starting to sour on these "do it all for me" usage scenarios for ai... especially for unfamiliar areas...
Agreed. Which is also odd, if you think about it. Surely with the amount of compute Anthropic and others have available, they could test each of the solutions in the SO data they surely have and rank them based on efficiency/elegance/other criteria and remove poor solutions from their training data.
Definitely not better off. SO was fairly mean spirited, but nowhere else has such a vast trove of high quality answers to common software problems been collected. SO likely trained many of these models with its answers, and I don't know what software development will look like when it dies.
No where else has such a vast trove of high quality answers been hidden because the question was closed as duplicate when someone else asked that question later.
I am thinking to make canned encyclopaedia of stackoverflow answers.
Claude/Grok/Gemini/Chatgpt answers are often so… how to say it… misleading? I have to stop the conversation as it leads nowhere (and it is not a skill issue :)
And, the AI trained on Stack Overflow. So if no one is posting new questions, and new answers. What will AI train on next, for the next thing.
Stack Overflow and Reddit are still getting threads. And as AI gets smarter, the questions will also expand.
your prompts, and the code you have it review.
Maybe.
I thought point was on Stack Overflow, there were community voting on 'best' answer.
If it is just me and the AI. Then the AI training data, is just whatever I approved the AI to do. Just my opinion.
We may need to create a community driven version of SO. Hard for it to be a successful business these days.
https://software.codidact.com/ was created after one of the many SO dramas. It doesn't come up in searches though and I didn't have reason to use it...
thanks, that's exactly what I was imagining.
What are you looking for and finding on stack overflow that isn't begginer to intermediate level?
agreed; sometimes the most useful info on SO wasn't in the replies, but the follow-up comments to the replies