Comment by porcoda
5 months ago
Not a big surprise once LLMs came along: stack overflow developed some pretty unpleasant traits over time. Everything from legitimate questions being closed for no good reason (or being labeled a duplicate even though they often weren’t), out of date answers that never get updated as tech changes, to a generally toxic and condescending culture amongst the top answerers. For all their flaws, LLMs are so much better.
Agreed. I personally stopped contributing to StackOverflow before LLMs, because of the toxic moderation.
Now with LLMs, I can't remember the last time I visited StackOverflow.
People in this thread are missing another key component in the decline of StackOverflow - the more experienced you become, the less useful it is.
The harder the problem, the less engagement it gets. People who spend hours working on your issue are rewarded with a single upvote. Meanwhile, "how do I concat a string" gets dozens or hundreds of upvotes.
The incentive/reward structure punished experienced folks with challenging/novel questions.
Pair that with the toxic moderation and trigger-happy close-votes, you get a zombie community with little new useful content.
Also: the bigger the corpus of already answered questions, it’s more likely that you can just look up an answer instead of asking.
Eventually SO becomes a site exclusively for lurkers instead of a platform for active participation
2 replies →
> People in this thread are missing another key component in the decline of StackOverflow - the more experienced you become, the less useful it is.
This is literally not true. The rate you learn and encounter new things depends on many things: you, your mood, your energy etc. But not on the amount of your experience.
> The harder the problem, the less engagement it gets. People who spend hours working on your issue are rewarded with a single upvote.
This is true, but not relevant, I don't think many people care. Some might, but not many.
8 replies →
Here's my brilliant idea: the longer it takes for an answer to be marked correct, or the more answers there are before one is marked correct, the more points that answer deserves.
5 replies →
Remember when the R developers would ask and answer their own basic questions about R, essentially building up a beginner tutorial on stack overflow? That was a cool time
2 replies →
This has been my experience.
My initial (most popular) questions (and I asked almost twice as many questions, as I gave answers) were pretty basic, but they started getting a lot more difficult, as time went on, and they became unanswered, almost always (I often ended up answering my own question, after I figured it out on my own).
I was pretty pissed at this, because the things I encountered, were the types of things that people who ship, encounter; not academic exercises.
Tells me that, for all the bluster, a lot of folks on there, don't ship.
LLMs may sometimes give pretty sloppy answers, but they are almost always ship-relevant.
That might be true on Stackoverflow but not on other network sites like Cross Validated, which was killed by splitting the community into multiple SE sites and longtime users quitting in protest over various policies and not being replaced.
7 replies →
Yeah, I think this is the real answer. I still pop into SO when in learning a new language or trip into new simple questions (in my case, how to connect and test a local server). But when you're beyond the weeds, SO is as best an oasis in the desert. Half the time a mirage, nice when it does help out. But rare either way.
I don't use LLMs eother. But the next generation might feel differently and those trends mean there's no new users coming in.
Wasn't there a "bounty" program where if it had a lot of views but no answers, the answer rewarded more internet ego points?
1 reply →
Maybe there's a key idea for something to replace StackOverflow as a human tech Q&A forum: Having a system which somehow incentivizes asking and answering these sorts of challenging and novel questions. These are the questions which will not easily be answered using LLMs, as they require more thought and research.
There is also github issues discussions now which also helped in asking these niche questions directly to the team responsible. I dont ask questions about a library on SO I just ask it on the github of the library and I get immediate answers
> the more experienced you become, the less useful it is
This is killer feature of LLMs - you will not became more experienced.
>toxic moderation and trigger-happy close-votes
>zombie community
Like Reddit post 2015.
2 replies →
Same here. I just didn't want to expend energy racing trigger happy mods. It was so odd, to this day remember vividly how they cleanup their arguments once proven wrong on the closing vote. Literally minutes before it would the close threshold.
Gen 0: expertsexchange.com, later experts-exchange.com (1996)
Gen 1: stackoverflow.com (2008)
Gen 2: chatgpt.com (2022, sort of)
Yahoo answers
Google answers
And the horrific Quora
3 replies →
>Gen 0: expertsexchange.com
No way.
1 reply →
And you can't delete your post when you realize how awful it was years later! That anti-information sticks around for ages. Even worse when there are bad answers attached to it, too.
Fun story: SO officially states comments are ephemeral and can be deleted whenever, so I deleted some of my comments. I was then banned. After my ban expired I asked on the meta site if it was okay to delete comments. I was banned again for asking that.
If you're talking about deleting questions, that's because deleting the question would delete everyone's answers that they potentially worked very hard on and which others might find useful. If you think the answers are bad you can always post your own competing answer.
2 replies →
You can’t delete anything here either… so make sure you don’t say anything awful.
1 reply →
The dumbest part of SO is how the accepted answer would often be bad, and sometimes someone had posted a better answer after the fact, and if the all-powerful moderators had the power to update it, they sure never did. Likewise, there were often better insights in comments. Apparently if you have the right mod powers, you can just edit an answer (such as the accepted one) to make it correct, but that always struck me as a bizarre feature, to put words in other people’s mouths.
I think overall SO took the gamification, and the “internet points” idea, way too far. As a professional, I don’t care about Reddit Karma or the SO score or my HN karma. I just wanted answers that are correct, and a place to discuss anything that’s actually interesting.
I did value SO once as part of the tedious process of attempting to get some technical problem solved, as it was the best option we had, but I definitely haven’t been there since 2023. RIP.
> took gamification way too far
I disagree, I always thought it SO did a great job with it. The only part I would have done differently would be to cap the earnable points per answer. @rndusr124 shouldn't have moderation powers just because his one and only 2009 answer got 3589 upvotes.
The same is true for reddit imo, it became impossible to post anything to a subreddit way before LLMs
Seemed like for every other question, I received unsolicited advice telling me how I shouldn't be doing it this way, only for me to have to explain why I wanted to do it this way (with silence from them).
This is called the XY problem https://meta.stackexchange.com/a/66378 . You ask for X, I tell you that what you really want is Y, I bully you, and I become more convinced that you and people that ask for X want Y.
Oh I love that game! (At least I think it's a game)
You ask how to do X.
Member M asks why you want to do X.
Because you want to do Y.
Well!? why do you want to do Y??
Because Y is on T and you can't do K so you need a Z
Well! Well! Why do you even use Z?? Clearly J is the way it is now recommended!
Because Z doesn't work on a FIPS environment.
...
Can you help me?
...
I just spent 15 minutes explaining X, Y and Z. Do you have any help?
...(crickets)
To be fair, asking why someone wants to do something is often a good question. Especially in places like StackOverflow where the people asking questions are often inexperienced.
I see it all the time professionally too. People ask "how do I do X" and I tell them. Then later on I find out that the reason they're asking is because they went down a whole rabbit hole they didn't need to go down.
An analogy I like is imagine you're organising a hike up a mountain. There's a gondola that takes you to the top on the other side, but you arrange hikes for people that like hiking. You get a group of tourists and they're all ready to hike. Then before you set off you ask the question "so, what brings you hiking today" and someone from the group says "I want to get to the top of the mountain and see the sights, I hate hiking but it is what it is". And then you say "if you take a 15 minute drive through the mountain there's a gondola on the other side". And the person thanks you and goes on their way because they didn't know there was a gondola. They just assumed hiking was the only way up. You would have been happy hiking them up the mountain but by asking the question you realised that they didn't know there was an easier way up.
It just goes back to first principles.
The truth is sometimes people decide what the solution looks like and then ask for help implementing that solution. But the solution they chose was often the wrong solution to begin with.
6 replies →
My heuristic is that if your interlocutor asks follow-up questions like that with no indication of why (like “why do you want to do X?” rather than “why do you want to do X? If the answer is Y, then X is a bad approach because Q, you should try Z instead”) then they are never going to give you a helpful answer.
How do I add a second spout to this can?
...
Well, the pump at the gas station doesn't fit in my car, but they sold me a can with a spout that fits in my car.
...
It's tedious to fill the can a dozen times when I just want to fill up my gas tank. Can you help me or not?
...
I understand, but I already bought the can. I don't need the "perfect" way to fill a gas tank, I just want to go home.
1 reply →
Tbf the problem there is probably FIPS more than anything else.
2 replies →
To avoid going insane the mindset should be to produce something useful for future readers.
Long before LLMs. Setting aside peak-COVID as a weird aberration, question volume has been in decline since 2014 or maybe 2016.
Stack Overflow would still have a vibrant community if it weren't for the toxic community.
Imagine a non-toxic Stack Overflow replacement that operated as an LLM + Wiki (CC-licensed) with a community to curate it. That seems like the sublime optimal solution that combines both AI and expertise. Use LLMs to get public-facing answers, and the community can fix things up.
No over-moderation for "duplicates" or other SO heavy-handed moderation memes.
Someone could ask a question, an LLM could take a first stab at an answer. The author could correct it or ask further questions, and then the community could fill in when it goes off the rails or can't answer.
You would be able to see which questions were too long-tail or difficult for the AI to answer, and humans could jump in to patch things up. This could be gamified with points.
This would serve as fantastic LLM training material for local LLMs. The authors of the site could put in a clause saying that "training is allowed as long as you publish your weights + model".
Someone please build this.
Edit: Removed "LLMs did not kill Stack Overflow." first sentence as suggested. Perhaps that wasn't entirely accurate, and the rest of the argument stands better on its own legs.
The fact that they basically stopped the ability to ask 'soft' questions without a definite answer made it very frustrating. There's no definitive answer to a question about best practices, but you can't ask people to share their experiences or recommendations.
2 replies →
Fixing loads of LLM-generated content is neither easy nor fun. You'll have a very hard time getting people to do that.
5 replies →
The community is not "toxic". The community is overwhelmed by newcomers believing that they should be the ones who get to decide how the site works (more charitably: assuming that they should be able to use the site the same way as other sites, which are not actually at all the same and have entirely different goals).
I don't know why you put "duplicates" in quotation marks. Closing a duplicate question is doing the OP (and future searchers) a service, by directly associating the question with an existing answer.
> Someone could ask a question, an LLM could take a first stab at an answer. The author could correct it or ask further questions, and then the community could fill in when it goes off the rails or can't answer.
Isn't this how Quora is supposed to operate?
2 replies →
Absolutely 100% this. I've used them on and off throughout the years. The community became toxic, so I took my question to other platforms like Reddit (they became toxic as well) and elsewhere.
Mind you, while I'm a relative nobody in terms of open source, I've written everything from emulators and game engines in C++ to enterprise apps in PHP, Java, Ruby, etc.
The consistent issues I've encountered are holes in documentation, specifically related to undocumented behavior, and in the few cases I've asked about this on SO, I received either no response and downvotes, or negative responses dismissing my questions and downvotes. Early on I thought it was me. What I found out was that it wasn't. Due to the toxic responses, I wasn't about to contribute back, so I just stopped contributing, and only clicked on an SO result if it popped up on Google, and hit the back button if folks were super negative and didn't answer the question.
Later on, most of my answers actually have come from Github,and 95% of the time, my issues were legitimate ones that would've been mentioned if a decent number of folks used the framework, library, or language in question.
I think the tl;dr of this is this: If you can't provide a positive contribution on ANY social media platform like Stack Overflow, Reddit, Github, etc. Don't speak. Don't vote. Ignore the question. If you happen to know, help out! Contribute! Write documentation! I've done so on more than one occasion (I even built a website around it and made money in the process due to ignorance elsewhere, until I shut it down due to nearly dying), and in every instance I did so, folks were thankful, and it made me thankful that I was able to help them. (the money wasn't a factor in the website I built, I just wanted to help folks that got stuck in the documentation hole I mentioned)
EDIT: because I know a bunch of you folks read Ars Technica and certain other sites. I'll help you out: If you find yourself saying that you are being "pedantic", you are the problem, not the solution. Nitpicking doesn't solve problems, it just dilutes the problem and makes it bigger. If you can't help, think 3 times and also again don't say anything if your advice isn't helpful.
It doesn't have anything to do with LLMs. It has to do with shifting one's focus from doing good things to making money. Joel did that, and SO failed because of it.
Joel promised the answering community he wouldn't sell SO out from under them, but then he did.
And so the toxicity at the top trickled down into the community.
Those with integrity left the community and only toxic, selfcentered people remained to destroy what was left in effort to salvage what little there was left for themselves.
Mods didn't dupe questions to help the community. They did it to keep their own answers at the top on the rankings.
How did Joel sell out? Curious as I’m not aware of any monetary changes. I watched Joel several times support completely brain dead policies in the meta discussions which really set the rules and tone. So my respect there is low.
They literally sold: https://arstechnica.com/gadgets/2021/06/stack-overflow-sold-...
He and Jeff made it abundantly clear their mission was to destroy the sex change site because that site was immoral for monetizing the benevolence of the community who answered the questions.
"Knowledge should be free" they said. "You shouldn't make money off stuff like this," they said.
Plenty of links and backstory in my other comments.
I also wonder if GitHub Discussions was also a (minor) contributing factor to the decline. I recall myself using GitHub Discussions more and more when it came to repo specific issues.
The timeline also matches:
https://github.blog/changelog/2020-12-08-github-discussions-...
https://github.blog/news-insights/product-news/github-discus...
Do we have any stats for the number of GitHub discussions created each month to compare to this?
It seemed to me that pre-llm, google had stopped surfacing stackoverflow answers in search results.
My memory is there were a spate of SO scraping sites that google would surface above SO and google just would not zap.
It would have been super trivial to fix but google didn’t.
My pet theory was that google were getting doubleclick revenue from the scrapers so had incentives to let them scrape and to promote them in search results.
I remember those too! There were seemingly thousands of them!
Reminds me of my most black-hat project — a Wikipedia proxy with 2 Adsense ads injected into the page. It made me like $20-25 a month for a year or so but sadly (nah, perfectly fairly) Google got wise to it.
2 replies →
Because nobody was clicking on them
> legitimate questions being closed for no good reason
They are closed for good reasons. People just have their own ideas about what the reasons should be. Those reasons make sense according to others' ideas about what they'd like Stack Overflow to be, but they are completely wrong for the site's actual goals and purposes. The close reasons are well documented (https://meta.stackoverflow.com/questions/417476) and well considered, having been exhaustively discussed over many years.
> or being labeled a duplicate even though they often weren’t
I have seen so many people complain about this. It is vanishingly rare that I actually agree with them. In the large majority of cases it is comically obvious to me that the closure was correct. For example, there have been many complaints in the Python tag that were on the level of "why did you close my question as a duplicate of how to do X with a list? I clearly asked how to do it with a tuple!" (for values of X where you do it the same way.)
> a generally toxic and condescending culture amongst the top answerers.
On the contrary, the top answerers are the ones who will be happy to copy and paste answers to your question and ignore site policy, to the constant vexation of curators like myself trying to keep the site clean and useful (as a searchable resource) for everyone.
> For all their flaws, LLMs are so much better.
I actually completely agree that people who prefer to ask LLMs should ask LLMs. The experience of directly asking (an LLM) and getting personalized help is explicitly the exact thing that Stack Overflow was created to get away from (i.e., the traditional discussion forum experience, where experts eventually get tired of seeing the same common issues all the time and all the same failures to describe a problem clearly, and where third parties struggle to find a useful answer in the middle of along discussion).
You seem to have filled this thread with a huge number of posts that try to justify SO's actions. Over and over, these justifications are along the lines of "this is our mission", "read our policy", "understand us".
Often, doing what your users want leads to success. Stamping authority over your users, and giving out a constant air of "we know better than all of you", drives them away. And when it's continually emphasized publicly (rather than just inside a marketing department) that the "mission" and the "policy" are infinitely more important than what your users are asking for, that's a pretty quick route to failure.
When you're completely embedded in a culture, you don't have the ability to see it through the eyes of the majority on the outside. I would suggest that some of your replies here - trying to deny the toxicity and condescension - are clearly showing this.
> Often, doing what your users want leads to success.
You misunderstand.
People with accounts on Stack Overflow are not "our users".
Stack Exchange, Inc. does not pay the moderators, nor high-rep community members (who do the bulk of the work, since it is simply far too much for a handful of moderators) a dime to do any of this.
Building that resource was never going to keep the lights on with good will and free user accounts (hence "Stack Overflow for Teams" and of course all the ads). Even the company is against us, because the new owners paid a lot of money for this. That doesn't change what we want to accomplish, or why.
> When you're completely embedded in a culture, you don't have the ability to see it through the eyes of the majority on the outside.
I am not "embedded in" the culture. I simply understand it and have put a lot of time into its project. I hear the complaints constantly. I just don't care. Because you are trying to say that I shouldn't help make the thing I want to see made.
> trying to deny the toxicity and condescension
I consider the term "toxicity" more or less meaningless in general, and especially in this context.
As for "condescension", who are you to tell me what I should seek to accomplish?
> "why did you close my question as a duplicate of how to do X with a list? I clearly asked how to do it with a tuple!" (for values of X where you do it the same way.)
This is a great example of a question that should not be closed as a duplicate. Lists are not tuples in Python, regardless of how similar potential answers may be.
I'm talking here about cases (which is basically all of them) where the first person to ask was simply needlessly specific. Or where the canonical has the list as an incidental detail and the next person insists that the answers won't work because this code has a tuple, you see, and doesn't see the merit in trying them.
If you imagine that the answer should be re-written from scratch to explain that the approach will be the same, you have fundamentally misunderstood the purpose of the site. Abstraction of contextually unimportant details is supposed to be an essential skill for programmers.
Oh yeah.
My favorite feature of LLMs, is the only dumb question, is the one I don't ask.
I guess someone could train an LLM to be spiteful and nasty, but that would only be for entertainment.
If you say the wrong thing to grok, it will go off on you. It's quite entertaining!
I suppose all sites that have a voting component run the risk of becoming unpleasant.
Hacker News, and we who frequent it, ought to have that in mind.
dang and the other HN moderators do a heroic job to set the tone, which has second- and third-order effects on behavior.
[flagged]
I think it has more to do with the fact that when you offer zero salary for moderators, you have to take what you can get, and it ain't good. I don't really see a connection to the voting mechanic.
It's also disconnected incentives. SO users get numbers to go up by taking moderation actions so of course they do that. Also you literally get banned from reviewing questions if you don't flag enough of them to be closed. These are incentives put in place by the SO company intentionally.
It's not like only slimy people get to use moderator tools like on Reddit, since you need a lot of reputation points you get by having questions and answers voted up. It's more like (1) you select people who write surface-level-good answers since that's what's upvoted, and they moderate with a similar attitude and (2) once you have access to moderator tools you're forced to conform with (1) or your access is revoked, and (3) the company is completely incompetent and doesn't give a shit about any of this.
Why do you think it makes a difference if they are paid or not? Or more to the point: what are you saying? That people have different standards when paid? That lack of remuneration justifies poor effort? Isn’t that a very transactional view of human interaction? Are we that transactional? Do we want this?
We’re talking about how communities can become toxic. How we humans sometimes create an environment that is at odds with our intentions. Or at least what we outwardly claim to be our intentions.
I think it is a bit sad when people feel they have to be compensated to not let a community deteriorate.
4 replies →
The irony is that the LLMs are trained on stack overflow and should inherit a lot of those traits and errors.
Yeah, but they don't inherit their rules and attitude.
Really, if we could apply some RLHF to the Stack Overflow community, it would be doing a lot better.
>For all their flaws, LLMs are so much better
But LLMs get their answers from StackOverflow and similar places being used as the source material. As those start getting outdated because of lack of activity, LLMs won't have the source material to answer questions properly.
I regularly use Claude and friends where I ask it to use the web to look at specific GitHub repos or documentation to ask about current versions of things. The “LLMs just get their info from stack overflow” trope from the GPT-3 days is long dead - they’re pretty good at getting info that is very up to date by using tools to access the web. In some cases I just upload bits and pieces from a library along with my question if it’s particularly obscure or something home grown, and they do quite well with that too. Yes, they do get it wrong sometimes - just like stack overflow did too.
The amount of docs that have a “Copy as markdown” or “Copy for AI” button has been noticeably increasing, and really helps the LLM with proper context.
Yeah that's a charitable way to phrase "perform distributed denial of service attacks". Browsing github as a human with their draconian rate limits that came about as a result of AI bots is fucking great.
2 replies →
StackOverflow answers are outdated. Every time I end up on that site these days, I find myself reading answers from 12 years ago that are no longer relevant.
I see plenty of old answers that are still very relevant. Suppose it depends on what language/tech tags you follow.
There have been many times I have seen someone complain on the meta site about answers being old and outdated, and then they give specific examples, and I go check them out and they're actually still perfectly valid.
Now they can read the documentation and code in the repo directly and answer based on that.
SO had answers that you couldn't find in the documentation and were you can't look in the source code.
If everything would be well documentated SO wouldn't have being as big as it was in the first place.
I think the industry is quickly moving to syntheticly derived knowledge, or custom/systematic knowledge production from humans.
That depends on what you mean by "came along". If you mean "once that everyone got around to the idea that LLMs were going to be good at this thing" then sure, but it was not long ago that the majority of people around here were very skeptical of the idea that LLMs would ever be any good at coding.
What you're arguing about is the field completely changing over 3 years; it's nothing, as a time for everyone to change their minds.
LLMs were not productified in a meaningful way before ChatGPT in 2022 (companies had sufficiently strong LLMs, but RLHF didn't exist to make them "PR-safe"). Then we basically just had to wait for LLM companies to copy Perplexity and add search engines everywhere (RAG already existed, but I guess it was not realistic to RAG the whole internet), and they became useful enough to replace StackOverflow.
I dont think this is true. People were skeptical of agi / better than human coding which is not the case. As a matter of fact i think searching docs was one of the first manor uses of llms before code.
That's because there has been rapid improvement by LLMs.
Their tendency to bullshit is still an issue, but if one maintains a healthy skepticism and uses a bit of logic it can be managed. The problematic uses are where they are used without any real supervision.
Enabling human learning is a natural strength for LLMs and works fine since learning tends to be multifaceted and the information received tends to be put to a test as a part of the process.
How can we be sure that LLMs won't start giving stale answers?
We can't. I don't think the LLMs themselves can recognize when an answer is stale. They could if contradicting data was available, but their very existence suppresses the contradictory data.
LLMs don't experience the world, so they have no reason a priori to know what is or isn't truthful in the training data.
(Not to mention the confabulation. Making up API method names is natural when your model of the world is that the method names you've seen are examples and you have no reason to consider them an exhaustive listing.)
They will, but model updates and competition help solve the problem. If people find that Claude consistently gives better/more relevant answers over GPT, for example, people will choose the better model.
The worst thing with Q/A sites isn't they don't work. It's that they there are no alternatives to stackoverflow. Some of the most upvoted answers on stackoverflow prove that it can work well in many cases, but too bad most other times it doesn't.
They still use the official documentation/examples, public Github Repos, and your own code which are all more likely to be evergreen. SO was definitely a massive training advantage before LLMs matured though.
LLMs are just statistics, eventually they kill themselves with feedback loop by consuming their own farts (literally)
all true, but i still find myself ask questions there after llm gave wrong answers and wasted my time
Yep, LLMs are perfect for the "quick buy annoying to answer 500 times" questions about writing a short script, or configuring something, or using the right combination of command line parameters.
Quicker than searching the entirety of Google results and none of the attitude.
Indeed. StackOverflow was by far the most unpleasant website that I have regularly interacted with. Sometimes, just seeing how users were treated there (even in Q&A threads that I wasn’t involved in at all) disturbed me so much it was actually interfering with my work. I’m so, so glad that I can now just ask an AI to get the same (or better) answers, without having to wade through the barely restrained hate on that site.
not only stackoverflow, but also reddit.com/r/aws reddit.com/r/docker reddit.com/r/postgresql all 3 of them have extremely toxic communities. ask a question and get downvoted instantly! Noo!! your job is to actually upvote the question to maximize exposure for the algorithm unless it is a really really stupid question that a google search could fix
> For all their flaws, LLMs are so much better.
For now. They still need to be enshitted.
Models are check-pointed. You can save one you like and use it forever.
You can save an open source + open weights model, which is frozen in time. That’s still very useful for some things but lacks knowledge of current data.
So we’ll end up with a choice of low-performing stale models or high-performing enshittified models which know about more current information.
2 replies →
Doesn't help when the ads are a layer above the model.
1 reply →
Yup. Like Claude 3 Opus.
Really? I thought you could only do that with open source models. Can you teach me how to checkpoint the current version of Claude Code so I can keep it as-is forever?
Yeah just wait for the ads
This change was happening well before LLMs. People were tired of being yelled at and treated poorly.
A cautionary tale for many of these types of tech platforms, this one included.