Comment by modeless
2 months ago
This is a cool idea. I would install a Chrome extension that shows a score by every username on this site grading how well their expressed opinions match what subsequently happened in reality, or the accuracy of any specific predictions they've made. Some people's opinions are closer to reality than others and it's not always correlated with upvotes.
An extension of this would be to grade people on the accuracy of the comments they upvote, and use that to weight their upvotes more in ranking. I would love to read a version of HN where the only upvotes that matter are from people who agree with opinions that turn out to be correct. Of course, only HN could implement this since upvotes are private.
The RES (Reddit Enhancement Suite) browser extension indirectly does this for me since it tracks the lifetime number of upvotes I give other users. So when I stumble upon a thread with a user with like +40 I know "This is someone whom I've repeatedly found to have good takes" (depending on the context).
It's subjective of course but at least it's transparently so.
I just think it's neat that it's kinda sorta a loose proxy for what you're talking about but done in arguably the simplest way possible.
I am not a Redditor, but RES sounds like it would increase the ‘echo-chamber’ effect, rather than improving one’s understanding of contributors’ calibration.
Echo chamber of rational, thoughtful and truthful speakers is what I’m looking for in Internet forums.
8 replies →
it depends on if you vote based on the quality of contribution to the discussion or based on how much you agree/disagree.
2 replies →
Echo chambers will always result on social media. I don't think you can come up with a format that will not result in consolidated blocs.
Reddit's current structure very much produces an echo chamber with only one main prevailing view. If everyone used an extension like this I would expect it to increase overall diversity of opinion on the site, as things that conflict with the main echo chamber view could still thrive in their own communities rather than getting downvoted with the actual spam.
3 replies →
More than having exact same system but with any random reader voting ? I'd say as long as you don't do "I disagree therefore I downvote" it would probably be more accurate than having essentially same voting system driven by randoms like reddit/HN already does
That assumes your upvotes in the past were a good proxy for being correct today. You could have both been wrong.
>This is a cool idea. I would install a Chrome extension that shows a score by every username on this site grading how well their expressed opinions match what subsequently happened in reality, or the accuracy of any specific predictions they've made.
Why stop there?
If you can do that you can score them on all sorts of things. You could make a "this person has no moral convictions and says whatever makes the number go up" score. Or some other kind of score.
Stuff like this makes the community "smaller" in a way. Like back in the old days on forums and IRC you knew who the jerks were.
That’s what Elon’s vision was before he ended up buying Twitter. Keep a digital track record for journalists. He wanted to call it Pravda.
(And we do have that in real life. Just as, among friends, we do keep track of who is in whose debt, we also keep a mental map of whose voice we listen to. Old school journalism still had that, where people would be reading someone’s column over the course of decades. On the internet, we don’t have that, or we have it rarely.)
I long had a similar idea for stocks. Analyze posts of people giving stock tips on WSB, Twitter, etc and rank by accuracy. I would be very surprised if this had not been done a thousand times by various trading firms and enterprising individuals.
Of course in the above example of stocks there are clear predictions (HNWS will go up) and an oracle who resolves it (stock market). This seems to be a way harder problem for generic free form comments. Who resolves what prediction a particular comment has made and whether it actually happened?
> Analyze posts of people giving stock tips on WSB, Twitter, etc and rank by accuracy.
Didn't somebody make an ETF once that went against the prediction of some famous CNBC stock picker, showing that it would have given you alpha in the past.
> seems to be a way harder problem for generic free form comments.
That's what prediction markets are for. People for whom truth and accuracy matters (often concentrated around the rationalist community) will often very explicitly make annual lists of concrete and quantifiable predictions, and then self-grade on them later.
You probably mean Inverse Cramer:
https://finbold.com/inverse-cramer-leaves-sp-nasdaq-and-dow-...
Cramer is the stock picker guy. There is a well known "Cramer Effect" or "Cramer Bounce" where the stock peaks then drops hard.
Makes for great pump n dump if you're day trading and willing to ride
https://www.investopedia.com/terms/c/cramerbounce.asp
long-term his choices don't do well, so the Inverse Cramer basically says "do the opposite of this goober" and has solid returns (sorta; depends a lot on methodology, and the sole hedgefund playing that strategy shutdown)
I ran across Sybil [1] the other day which tries to offer a reputation score based on correct predictions in prediction markets.
[1]: https://sybilpredicttrust.info/
Out of curiosity, I built this. I extended karpathy's code and widened the date range to see what stocks these users would pick given their sentiments.
What came back were the usual suspects: GLP-1 companies and AI.
Back to the "boring but right" thesis. Not much alpha to be found
I like the idea and certainly would try it. Although I feel in a way this would be an anti-thesis to HN. HN tries to foster curiosity, but if you're (only) ranked by the accuracy of your predictions, this could give the incentive to always fall back to a save and boring position.
I think the most interesting predictions are the ones that sound bold and even a little bit insane at the time. I think a lot more of the people who were willing to say saying "ASI will kill us all" 20+ years ago, because they were taking a risk (and routinely ridiculed for it).
Even today, "ASI will kill us all" can be a pretty divisive declaration - hardly safe and boring.
From the couple of threads I clicked, it seemed like this LLM-driven analysis was picking up on that, too: the top comments were usually bold, and some of the worst-rated comments was the "safe and boring" declaration that nothing interesting ever really happens.
The problem seems underspecified; what does it mean for a comment to be accurate? It would seem that comments like "the sun will rise tomorrow" would rank highest, but they aren't surprising.
just because an idea is qualitative doesn't mean its invalid
Didn't Slashdot have something like the second point with their meta-moderation, many many years ago?
Sorta.
IIRC, when comment moderation and scoring came to Slashdot, only a random (and changing) selection of users were able to moderate.
Meta-moderation came a bit later. It allowed people to review prior moderation actions and evaluate the worth of those actions.
Those users who made good moderations were more likely to become a mod again in the future than those who made bad moderations.
The meta-mods had no idea whose actions they were evaluating, and previous/potential mods had no idea what their score was. That anonymity helped keep it honest and harder to game.
It's still that way today: if you're active, you'll be randomly given 5 moderation points occasionally, and they expire after a few days. So you have to decide which threads and comments are worth spending a "moderation point" on
1 reply →