Comment by louison11
1 year ago
My heart goes to the stack overflow community which has always been very kind and helpful, essentially working for free. As a self-taught developer since the age of 8, I literally grew up learning how to code through SO, asking hundreds of questions and answering many more. So many bugs that would take 2-3 days to fix would eventually find their answer through it. But now ChatGPT does that in minutes… so it’s for the best!
The presumption is that things will improve over time, but the big difference in my experience is the assistance I got from SO _worked_ the vast majority of the time, whereas various LLMs I have used generate unusable, misleading, or unreliable results pretty regularly, increasing as complexity or rarity arises. As human-driven knowledge bases backed by actual experience are replaced by inference from models who rely on such inputs, I am concerned about the medium to long term impact. A lot of people grew frustrated with SO for various reasons and went back to unhelpful behaviors that SO resolved at its zenith (rather than dead ends and flame wars in newslists and irc channels, they do it in random subreddits and discord servers instead). Now what if we circle back after degenerative LLM experiences only to find there’s nothing to circle back on?
Personally, I am getting extremely tired of ChatGPT hallucinating npm packages that don't exist, or package imports that do not exist
Those hallucinated libraries exist in the future where the LLM lives and works. You just haven't caught up in time yet. ;-)
3 replies →
But then you can just ask it to write that missing library! Some day in the future you can probably ask it to author the whole package and publish it itself.
"Oh sorry, that package doesn't exist yet, but it ought to. One moment... Ok, try installing it now."
1 reply →
Get a good service. Running it on your old gaming machine?
2 replies →
I think this is the first time I’ve ever heard someone describe the stack overflow community as “kind”. Usually it’s the exact opposite: “I asked a question and just got 30 questions asking why I’m trying to do this” or “my question was closed for seemingly no reason”.
It’s literally the most blunt and aggressive website I’ve ever been on that wasn’t a straight-up troll site like 4-Chan.
I answer questions for a few tags, know the styles of the other people who answer on those tags, and I do consider my fellow answerers as kind. All of them.
That doesn't mean that you will think we're kind to you personally. We're there to build a searchable Q&A knowledge base and spread knowledge. Some people who ask questions misunderstand and think we're there to help them, personally. To work for free for that single person, and we're not there for that. We write answers for the tens, hundreds, thousands of people who will search for it.
Askers who misunderstand will come across as overly entitled.
In terms of practical effects: People who misunderstand don't tag their questions, or tag them incorrectly. They post screenshots full of text. They don't look for similar older questions in the existing knowledge base, or they insist that even slightly different questions are significantly different. All rather offputting, and often puzzling. How can you ask for a subject expert's help and simultaneously insist that you know better than the expert whether your question is a duplicate of another?
Reading between the lines, this is extremely telling. Of course, nobody is a villain in their own story. Members of online communities who drive others away are often just simply blind to the impact they are having - an existential impact in the case of Stack Overflow sadly.
8 replies →
How can you ask for a subject expert's help and simultaneously insist that you know better than the expert whether your question is a duplicate of another?
this is easy part, I read the answer of that “duplicate” and it was not applicable to my problem :)
11 replies →
Seems to me like you're part of the problem. Of course it's not my problem anymore because I no longer contribute there.
10 replies →
If that's you're goal, you're going about it the wrong way. Thank you for introducing yourself and your fellow answerers. Let me introduce myself and my fellow questioners. I have a deadline and a problem. I've already spent 5 hours researching why what should work - according the the documentation and the conventional advice - doesn't. I've searched many sources, including SO. I've seen some articles which might have answered my problem. Tried the suggestions, but no joy. So now I'm six hour in, and my deadline is looming. It's probably around 1:00am. Between 1 and 2 I type up my problem and submit it to SO. I'm hopeful that perhaps in the morning someone who has successfully worked through my problem will have contributed a solution.
9:00am, I check SO. My reputation has decreased by 8 points, a number of self-styled enforcers have left negative comments comparing my issue to other issues which bear a superficial similarity to my posting, and my posting has been closed.
I'm not the most powerful contributor but over several years I've achieved upwards of 1,000 points. So I am by no means a nudnick. I've posted some good ones and I've helped some of my peers along the way. But recently, my experience has devolved to the point where the experience I describe above is the rule, rather than the exception. And when I tried to have the discussion we are having now, on the stack overflow meta site, your fellow enforcers shut down the discussion and deleted the posting. So I left. And now we can have the conversation here.
You can have all the justifications in the world for your approach, and you don't need to keep the audience you don't want. But if those of us voicing our displeasure here, are not simply a few malcontents, but a significant chunk of your former user base, you might want to look inward, and at the same time ask with a certain measure of humility - what are we doing wrong and how can we improve?
For starters, if you want a questioner to improve their posting or you have questions about why they posted, is it necessary to start off by immediately deducting from the poster's reputation? Ask your question, make your point, give the poster the opportunity to remediate or show you why you're the one who's off base (did you ever consider that possibility?) before decreasing someone's reputation.
Stop dehumanizing your knowledge base. Your resistance to AI is somewhat ironic, given all the effort you've devoted toward eliminating all courtesy and gratitude from your knowledge base. Do you want humans communicating on your platform? Let them. Perhaps after a question has been asked, answered, let the posting remain dormant for 30 days and then have some AI process go ahead and scrub the posting. Don't ding people for saying please and thank you and expect them to like you for it.
Just for starters. For now, I'm out of there. Change your game, maybe I'll be back one day.
2 replies →
There was a time when it was really good. Like legitimately good and useful. But over time it ended up becoming exactly what you describe. But there are still countless examples of the usefulness of SO in Google results. I stopped asking questions in 2012 and stopped answering questions in 2015. Before that though, it was a very useful tool.
Question closed; here's a link to another one that sounds vaguely related but doesn't actually address your problem.
But seriously, I'd love to see some sentiment analysis of the SO corpus classifying tone by tag.
I've closed like that. One asker complained that his question about base64 encoding in one language was not like the duplicate I identified because the language was another one. "Vaguely related", he thought, but he asked precisely because he didn't know.
4 replies →
"Having your problem addressed" is not a valid reason to post on Stack Overflow. You are expected before posting to have done enough analysis to the point where if your question is answered, you can solve the underlying problem yourself. When you are linked to a duplicate, it's because the person doing so believes in good faith that, to the extent that you have a question that meets the site's standards, answers to the other question will answer yours as well. This also means you are responsible for overlooking irrelevant details, reading the answers, making your own attempts to apply them, etc.
If the other question is actually different, you are expected to edit the question to make this clear - not by adding an "Edit:" section like in a forum post, but by fixing the wording such that it's directly clear what you're looking for and how it's different. This might mean fixing your specification of input or desired output.
It's difficult sometimes, and curators do make mistakes. Most frustratingly, it's entirely possible for two completely different problems to be reasonably described with all the same keywords. I personally had a hell of a time disentangling https://stackoverflow.com/questions/9764298 from https://stackoverflow.com/questions/18016827, while also explaining that https://stackoverflow.com/questions/6618515 really is the same as the first problem despite different phrasing.
But curators much more often get it right. Not only that, a few of us go out of our way to create artificial Q&A (https://meta.stackoverflow.com/questions/426205) for beginner issues that beginners never know how to explain, and put immense effort into both the question and answer. Some popular examples in the Python tag:
"I'm getting an IndentationError (or a TabError). How do I fix it?" (https://stackoverflow.com/questions/45621722) was written to replace "IndentationError: unindent does not match any outer indentation level, although the indentation looks correct" (https://stackoverflow.com/questions/492387) and a few others, with reasoning stated there.
"Asking the user for input until they give a valid response" (https://stackoverflow.com/questions/23294658)
"Why does "a == x or y or z" always evaluate to True? How can I compare "a" to all of those?" (https://stackoverflow.com/questions/20002503) was written largely as an alternative to the organic "How to test multiple variables for equality against a single value?" (https://stackoverflow.com/questions/15112125) after the latter was found not to help beginners very well (the original example was quite unclear, although it's since been improved).
26 replies →
Agree! A decade ago it wasn't like this. But it has devolved into a community of vandals who seem to take glee in criticizing the manner in which a question is asked, rather than contributing a solution.
> vandals
How is it vandalism to enforce a quality standard?
> who seem to take glee
Why do you imagine so?
4 replies →
Yep, believe it's a direct result of Atwood's iron-fisted no-bullshit policy. To some extent it is great... don't want it turning into Yahoo Answers, do we? I think folks forget about that part.
But, as you mention they just went too damn far with the medicine.
No, you can't fix this misspelling, isn't there something else (with more characters) that you can improve as well? WTF, for realz? :-/
>No, you can't fix this misspelling, isn't there something else (with more characters) that you can improve as well? WTF, for realz? :-/
I agree this complaint is legitimate. The problem is that the system expects unprivileged users to have their edits reviewed by three privileged users in a queue (so that people actually pay attention and vandalism doesn't just go unnoticed for months), so this is meant to limit the drain on that resource.
You may be interested in my answer to "Reviewer overboard! Or a request to improve the onboarding guidance for new reviewers in the suggested edits queue" on the meta site (https://meta.stackoverflow.com/a/420357/523612).
Mixed feelings on SO. It was helpful, but it was also a website you dread having to post on because it was filled with the most intolerable people of the internet who you just had to endure abuse from if you wanted help.
Now chatGPT gives you the same help without the abuse.
The next AI totally needs to be more snarky to make it feel more like we're dealing with actual "thinksperts", people that think they are experts even if their answers are demonstrably wrong.
"You are Comic Book Guy from The Simpsons. You are very knowledgeable in the _____ language; in fact, you believe you are the foremost expert on it. You have taken time from your busy schedule to help the unwashed masses by answering the following question..."
2 replies →
> But now ChatGPT does that in minutes
But it's trained on stackoverflow data? What happens in a few years when the data gets more and more outdated? Where will it get its knowledge then?
They're learning from working code in GitHub, IDE "co-pilots"...
But a priori you don't know if the code you find on Github is "good", plus it doesn't come with a handy explanation. The quality of the data is much, much worse.
1 reply →
It will steal our own data and we'll have a big "oopsie! didn't mean to!" moment 5-10 years after.
My point is that there won't even be any data to steal! The novel human-written and human-rated answers just won't exist anymore. Where will it get its answers on C++26 features from? Not the non-existing StackOverflow, that's for sure.
1 reply →
Why does any LLM need new information to do fundamentally the same thing?
And what makes the data outdated? New code? It can train on that. That, or there is simply nothing new to learn, just new ways to express the same thing.
> Why does any LLM need new information to do fundamentally the same thing?
What makes you think we will be doing fundamentally the same thing in the future? Language grow and change, systems change, operating systems change, hardware and specs change..
Nothing in computing is ever static.
Are you talking about Stackoverflow? Every time I asked a detailed question it would be closed within minutes.
I'm not surprised it's on the way out.
I've asked dozens of questions on SO, and never had a single one closed. I hear your sentiment often, but have no idea whether my experience or yours is more common.
I've had 3 deleted by Community bot as abandoned, but since they were over a year old when that happened, I couldn't care less.
I've got valuable advice from SO over the years. There's overlap with LLMs, sure, but it's frequent to have questions that have no answers published anywhere on the web; SO brings people who know out of the woodwork, who create an explanation that didn't exist before. A couple days ago, someone in retrocomputing got to bank-switch a 1983 Radio Shack box... that kind of stuff wasn't published anywhere, until a guy who used to write games for that box answered that question on SO.
Until there is a radically new version of {popular programming language} with breaking changes and no new and correct answers to train on.
These models can figure out syntax and language features they haven’t seen before. Try it with a few code snippets of your own made-up language. It’s a little freaky.
They can implicitly assume that your made-up language is designed to be easy to use by native language speakers, and thus apply their existing understanding of "code" to it, sure.
1 reply →
They can indeed, but 1) this takes up an inordinate amount of context, and 2) the more you force model to think about that, the less effective it is at actually writing code.
The point of an LLM is that it can take your problem as input, along with answers to previously asked questions (perhaps implicit) in its training data, and attempt to synthesize a solution to a problem as output. Here, a "question" is something that can be found with a search engine - something that directly presents the/a crux of an issue, which is identified after a debugging session (for a problem in existing code; for projects still in the design phase, a how-to question emerges after coming up with user stories and breaking tasks down into their logical steps). The point is that the question can be relevant to many different people who have written different code which encounters different problems - all caused fundamentally by the same conceptual misunderstanding (or non-understanding).
Stack Overflow is explicitly not designed or intended to solve problems or do the decomposition of the problem for you, nor the synthesis of answers. Because the result would never be useful to anyone else. The entire point is to have something searchable, and to allow answer-writers to keep their explanations DRY.
This has spectacularly failed, because no matter how frustrated people get with traditional discussion forums (https://www.explainxkcd.com/wiki/index.php/979:_Wisdom_of_th... among many other typical complaints), they apparently are much more suited to human nature.
Heaven knows how Wikipedia managed to avoid devolving into "Quora but even worse because you can scribble over someone else's post".
Closing this as its a statement not a question
I moved this over to the Stackoverflow Will Be Missed thread. It does not belong in the Let's Dance On Stackoverflow's Demise thread.
I just hope that we can continue to find sources of high quality training data like SO. If people don't publish their mutual learnings somewhere then there's no data to train on.
Ironically, when someone first asked the Meta SO community if ChatGPT could ever become a threat, it was downvoted with prejudice:
https://meta.stackexchange.com/questions/384355/could-chatgp...
It was downvoted because the entire purported "threat" is based on a misconception.
It is not relevant to SO whether an LLM can provide personalized help, write with any particular tone, answer promptly, accept every input prompt as valid and try to make sense of it, discuss back and forth to figure out a problem, etc. Because Stack Exchange is explicitly and by design not for those things.
But also, downvotes work differently on meta anyway, and the community there generally takes a negative view of LLMs. Because, again, the point of SO is for the answer to come from a human expert, and be verified against subject matter expertise rather than simply being evaluated for coherence or generally sounding appropriate in context.