Comment by etamponi
5 months ago
I spent the last 14 days chasing an issue with a Spark transform. Gemini and Claude were exceptionally good at giving me answers that looked perfectly reasonable: none of them worked, they were almost always completely off-road.
Eventually I tried with something else, and found a question on stackoverflow, luckily with an answer. That was the game changer and eventually I was able to find the right doc in the Spark (actually Iceberg) website that gave me the final fix.
This is to say that LLMs might be more friendly. But losing SO means that we're getting an idiot friendly guy with a lot of credible but wrong answers in place of a grumpy and possibly toxic guy which, however, actually answered our questions.
Not sure why someone is thinking this is a good thing.
What I always appreciate about SO is the dialogue between commenters. LLMs give one answer, or bullet points around a theme, or just dump a load of code in your IDE. SO gives a debate, in which the finer points of an issue are thrashed out, with the best answers (by and large) floating to the top.
SO, at its best, is numerous highly-experienced and intelligent humans trying to demonstrate how clever they are. A bit like HN, you learn from watching the back and forth. I don't think this is something that LLMs can ever replicate. They don't have the egos and they certainly don't have the experience.
Whatever people's gripes about the site, I learned a hell of a lot from it. I still find solutions there, and think a world without it would be worse.
The fundamental difference between asking on SO and asking an LLM is that SO is a public forum, and an LLM will be communicated with in private. This has a lot of implications, most of which surround the ability for people to review and correct bad information.
The other major benefit of SO being a public forum is that once a question was wrestled with and eventually answered, other engineers could stumble upon and benefit from it. With SO being replaced by LLMs, engineers are asking LLMs the same questions over and over, likely getting a wide range of different answers (some correct and others not) while also being an incredible waste of resources.
Surely the fundamental difference is one asks actual humans who know what's right vs statistical models that are right by accident.
6 replies →
SO also isn't afraid to tell you that your question is stupid and you should do it a better way.
Some people take that as a personal attack, but it can be more helpful than a detailed response to the wrong question.
The problem is the people who decide which questions are stupid are misaligned with the site's audience.
> What I always appreciate about SO is the dialogue between commenters.
Stack Overflow is explicitly not for "dialogue", recent experiments (which are generally not well received by the regulars on the meta site) notwithstanding. The purpose of the comments on questions is to help refine the question and ensure it meets standards, and in some cases serve other meta purposes like pointing at different-but-related questions to help future readers find what they're looking for. Comments are generally subject to deletion at any time and were originally designed to be visually minimal. They are not part of the core experience.
Of course, the new ownership is undoing all of that, because of engagement metrics and such.
Heh, OK, dialogue wasn't the right word. I am a better informed person by the power of internet pedantry.
> I don't think this is something that LLMs can ever replicate. They don't have the egos and they certainly don't have the experience
Interesting question - the result is just words so surely a LLM can simulate an ego. Feed it the Linux kernel mailing list?
Isn’t back and forth exactly what the new MoE thinking models attempt to simulate?
And if they don’t have the experience that is just a question of tokens?
SO was somewhere people put their hard won experience into words, that an LLM could train on.
That won't be happening anymore, neither on SO or elsewhere. So all this hard won experience, from actually doing real work, will be inaccessible to the LLMs. For modern technologies and problems I suspect it will be a notably worse experience when using an LLM than working with older technologies.
It's already true for example, when using the Godot game engine instead of Unity. LLMs constantly confuse what you're trying to do with Unity problems, offer Unity based code solutions etc.
> Isn’t back and forth exactly what the new MoE thinking models attempt to simulate?
I think the name "Mixture of Experts" might be one of the most misleading labels in our industry. No, that is not at all what MoE models do.
Think of it rather like, instead of having one giant black box, we now have multiple smaller opaque boxes of various colors, and somehow (we don't really know how) we're able to tell if your question is "yellow" or "purple" and send that to the purple opaque box to get an answer.
The result is that we're able to use less resources to solve any given question (by activating smaller boxes instead of the original huge one). The problem is we don't know in advance which questions are of which color: it's not like one "expert" knows CSS and the other knows car engines.
It's just more floating point black magic, so "How do I center a div" and "what's the difference between a V6 and V12" are both "yellow" questions sent to the same box/expert, while "How do I vertically center a div" is a red question, and "what's the most powerful between a V6 and V12" is a green question which activates a completely different set of weights.
I don't know if this is still the case but back in the day people would often redirect comments to some stackoverflow chat feature, the links to which would always return 404 not found errors.
This comment and the parent one make me realize that people who answer probably value the exchange between experts more than the answer.
Perhaps the antidote involves a drop of the poison.
Let an LLM answer first, then let humans collaborate to improve the answer.
Bonus: if you can safeguard it, the improved answer can be used to train a proprietary model.
> This comment and the parent one make me realize that people who answer probably value the exchange between experts more than the answer.
I'm more amused that ExpertsExchange.com figured out the core of the issue, 30 years ago, down to their site's name.
You can ask an LLM to provide multiple approaches to solutions and explore the pros and cons of each, then you can drill down and elaborate on particular ones. It works very well.
There are so many "great" answers on StackOverflow. Giving the why and not just the answer.
It's flat wrong to suggest SO had the right answer all the time, and in fact in my experience for trickier work it was often wrong or missing entirely.
LLMs have a better hit rate with me.
The example wasn't even finding a right answer so I don't see where you got that..
Searching questions/answers on SO can surface correct paths on situations where the LLMs will keep giving you variants of a few wrong solutions, kind of like the toxic duplicate closers.. Ironically, if SO pruned the history to remove all failures to match its community standards then it would have the same problem.
"But losing SO means that we're getting an idiot friendly guy with a lot of credible but wrong answers in place of a grumpy and possibly toxic guy which, however, actually answered our questions."
> "actually answered our questions."
Read carefully.
5 replies →
It entirely depends on the language you were using. The quality of both questions and answers between e.g. Go and JavaScript is incredible. Even as a relative beginner in JS I could not believe the amount of garbage that I came across, something that rarely happened for Go.
No point in arguing with people who bring a snowball into Congress to disprove global warming.
[flagged]
I mean when I was pure SO debugging I would say a solid 70% of the responses did not work. Different stack different bug ect.
LLMs even if they can't guess the right answer can search the documentation really well.
[flagged]
2 replies →
Because what you’re describing is the exception. Almost always with LLM’s I get a better solution, or helpful pointer in the direction of a solution, and I get it much faster. I honestly don’t understand anyone could prefer Google/SO, and in fact that the numbers show that they don’t. You’re in an extreme minority.
> But losing SO means that we're getting an idiot friendly guy with a lot of credible but wrong answers in place of a grumpy and possibly toxic guy which, however, actually answered our questions.
Which by the way is incredibly ironic to read on the internet after like fifteen years of annoying people left and right about toxic this and toxic that.
Extreme example: Linus Torvalds used to be notoriously toxic.
Would you still defend your position if the “grumpy” guy answered in Linus’ style?
> Would you still defend your position if the “grumpy” guy answered in Linus’ style?
If they answered correctly, yes.
My point is that providing _actual knowledge_ is by itself so much more valuable compared to _simulated knowledge_, in particular when that simulated knowledge is hyper realistic and wrong.
Sadly, an accountable individual representing an organization is different from a community of semi-anonymous users with a bunch of bureaucracy that can't or doesn't care about every semis anonymous user
You still get the same thing though?
That grumpy guy is using an LLM and debugging with it. Solves the problem. AI provider fine tunes their model with this. You now have his input baked into it's response.
How you think these things work? It's either a human direct input it's remembering or a RL enviroment made by a human to solve the problem you are working on.
Nothing in it is "made up" it's just a resolution problem which will only get better over time.
How does that work if there's no new data for them to train on, only AI slurry?
I'm hoping increasing we'll see agents helping with this sort of issue. I would like an agent that would do things like pull the spark repo into the working area and consult the source code/cross reference against what you're trying to do.
Once technique I've used successfully is to do this 'manually' to ensure codex/Claude code can grep around the libraries I'm using
Q&A isn't going away. There's still GitHub Discussions.