oh boy I wrote one of these many years ago for HN.
Within like an hour or two pg emailed me asking me to stop. I didn't know it at the time, but HN was being run on a rusty potato and scraping the homepage every 5 or 10 seconds was causing significant load.
Afaict, HN is still running on a rusty potato. The software's written well, so it doesn't need to run on more than that. (What's going to happen to it? Someone links to it from HN?)
(just making sure it's obvious - I didn't create this site! all credits to @jerbear4328 who is here on HN - I'll email them to let them know it's trending) :-)
Any tips on respectfully crawling HN so you don’t get throttled? I had an application idea that could not be served by the API (need karma values) so I started to write code to scrape but got rate limited pretty quickly.
I tried this, but it required making a request for every comment and would probably call for a backend, wheras this can run just off of the Firebase websocket stream on a static HTML file.
If you want a live version of most of the site (including tracking new comments in items you've already looked at), I made this to improve my React skills when the HN API first came out:
I just wish there was a dark mode.. Let me install night reader to see how well it looks :p
Edit: would've loved if there was a way to sign up or make comments from there, oh well :< I wish there was, I am not sure if that's possible tho but I hope it could be.
I checked the github repo and I found out that its a single static html page and ahh I forgot to see that its hosted on github.io as well because of their static hosting, somehow I didn't see it!!
I know they get an alert whenever a comment is flagged. I assume they get an alert whenever certain "problematic" accounts post. Maybe they get an alert whenever the flamewar detector goes off? This forum is written in a Lisp so they can probably hotpatch whatever code they want and add whatever filters they want.
But dang has said numerous times that he doesn't read everything, that would probably not even be feasible.
The question is, YC being what it is, are they using LLMs to automate moderation or do sentiment analysis?
Another great use of the HN API! It would be nice to filter it somehow to threads where I have commented, sometimes I don't notice until days later that someone replied to me.
Didn't know about this! Thanks, I just wish something like this was on signal or discord but maybe that's because I don't operate much mail and I really like signal...
But anyways, just as a headsup for a moment I was confused that I wasn't getting the verification mail and I was literally going to comment it but then I got the mail just in time, so maybe to anyone out there, maybe be a bit patient as its worth it (or it could totally just be an issue from my side as I spammed it twice or maybe it got congested I am not sure)
EDIT: It wasn't that they sent me 2 mails but rather that they sent me 1 mail and I clicked on it and then got another and I thought it was because of me spamming verification twice but it turns out that they sent me a mail because someone had responded to me :)
a very very quick copy/paste of 73 comments there - fed to claude with a basic "characterize+count comments" give me 70% neutral/technical, 14% positive, 16% negative.
claude also said "The negative comments tend to be more substantive critiques of systems or research approaches rather than personal attacks." - so... that's good? :-)
I've done it before on large threads. Neutral sentiment tends to dominate at roughly 50% but negative sentiment is much higher than positive sentiment. Negative sentiment is also higher if you just take the highest ranked comments.
It's actually quite stark looking at it this way and makes me question how healthy it is to use this site as often as I do. Fully aware that we're also piling on the negative sentiment.
Nothing actually gets deleted on HN, you can turn on "showdead" in your options to see flagged and dead comments. None of that gets sent with the API, though.
There is an API endpoint called "updates" but it's incredibly vague about what it actually shows, or how persistent it is. It might show edits.
Pretty cool! I must manually refresh to see new posts. Implementing real-time updates (e.x. via WebSocket or Server-Sent Events) would significantly improve usability.
It sounds cool, but for usability its not great. Think about how reddit recalculates the results order as you're paging through, and you see items from the previous page show up on the next. Now imagine that happening in realtime. Maybe there's a link you want to read, but you get pulled away for 10 minutes. By the time you get back the link is higher or lower, or may be completely missing.
That is a good point, however there are design solutions around it. For example:
- Poll every X seconds instead of real-time
- Enable user to toggle real-time mode
- Load new posts in with a "+" button at the top to fetch the latest posts (like Twitter)
It (hopefully) took exactly 30 seconds, the page delays every item until 30 seconds after its posted date. It doesn't poll HN's server, it opens a websocket to the official HN Firebase, and without the delay, items appear in large chunks. I'm pretty sure the HN server syncs with Firebase every 30 seconds, so this is as fast as it can go while still being accurate.
It works, but I'd advise about using the same branding. Consider making it significantly different than the website's comments page to avoid any issues, since it's not part of the webapp nor sanctioned.
Okay, maybe you could explain why copying the look of a product to look exactly like the said product, without it being warranted is a worth ignoring and funny?
oh boy I wrote one of these many years ago for HN.
Within like an hour or two pg emailed me asking me to stop. I didn't know it at the time, but HN was being run on a rusty potato and scraping the homepage every 5 or 10 seconds was causing significant load.
Haha, my version makes a websocket connection to the official Firebase that the HN servers already send everything to, so it is zero extra load on HN
Yours is the MUCH better approach. When I did it, no api!
That sounds interesting, are there any public details on this? Is it https://github.com/HackerNews/API ?
1 reply →
Afaict, HN is still running on a rusty potato. The software's written well, so it doesn't need to run on more than that. (What's going to happen to it? Someone links to it from HN?)
less rusty as of ~sep 2024 :-) https://news.ycombinator.com/item?id=44099006
10 replies →
A LISPy potato, right?
Source code: https://github.com/jerbear2008/hn-live
And live firebase source/api (run by HN) used for this: https://github.com/HackerNews/API
(just making sure it's obvious - I didn't create this site! all credits to @jerbear4328 who is here on HN - I'll email them to let them know it's trending) :-)
Ha amazing that this is just an HTML file
Very cool! Thanks for sharing!
(edit: terminal version available?)
I'm not the author of this live feed site, but I did just find
https://github.com/ggerganov/hnterm
from the author of llama.cpp (!!).
it also has a beautiful web version thanks to emscripten: https://hnterm.ggerganov.com/
It would be great to have the "on: x y z thread" field included.
This is literally the only thing that seems missing- Stupendous work!
https://jaytaylor.github.io/hn-live2
Enjoy!
I did this too: https://hn.hotgarba.ge/
Edit: Through my own tool I see this comment got insta-marked as [dead], rude.
I wonder if the domain is banned for some reason. Another comment of yours is dead, which includes a link to the same domain.
https://hotgarba.ge/
(Edit: Yep, this comment is dead on arrival.)
Love the domain lol
Cool idea.
Separately I didn’t know posts/submissions could be instantly flagged dead and I didn’t know there are decade+ old bots still configured to spam this site https://news.ycombinator.com/submitted?id=VivaTechnics
(2024) Show HN from the dev https://news.ycombinator.com/item?id=39509910
Cool!
Would be slightly more contextual if the title of the original post was displayed.
The HN/Firebase API doesn't make this easy. For https://hnstream.com I ended up crawling items to find the article.
Any tips on respectfully crawling HN so you don’t get throttled? I had an application idea that could not be served by the API (need karma values) so I started to write code to scrape but got rate limited pretty quickly.
1 reply →
The comments don't even have a thread ID?
5 replies →
I tried this, but it required making a request for every comment and would probably call for a backend, wheras this can run just off of the Firebase websocket stream on a static HTML file.
I found it a fun game to guess what I thought the post was about.
If you want a live version of most of the site (including tracking new comments in items you've already looked at), I made this to improve my React skills when the HN API first came out:
https://insin.github.io/react-hn
Wow it looks really cool, I think I might enjoy actually using it as the main way to view the site
I was going to say the same thing too!
I just wish there was a dark mode.. Let me install night reader to see how well it looks :p
Edit: would've loved if there was a way to sign up or make comments from there, oh well :< I wish there was, I am not sure if that's possible tho but I hope it could be.
I checked the github repo and I found out that its a single static html page and ahh I forgot to see that its hosted on github.io as well because of their static hosting, somehow I didn't see it!!
Static hosting is super cool and I am going to tinker with it to have monospace-web instead for something like this https://owickstrom.github.io/the-monospace-web/
Thanks a lot for creating this! Appreciate it, I haven't read the source code but I am going to read it oneday (If I don't procastinate lol)!
Have a nice day!
I often wonder if dang and tomhow have something like this running.
I wonder to what extent they read everything which is posted here.
Not even close.
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
1 reply →
I know they get an alert whenever a comment is flagged. I assume they get an alert whenever certain "problematic" accounts post. Maybe they get an alert whenever the flamewar detector goes off? This forum is written in a Lisp so they can probably hotpatch whatever code they want and add whatever filters they want.
But dang has said numerous times that he doesn't read everything, that would probably not even be feasible.
The question is, YC being what it is, are they using LLMs to automate moderation or do sentiment analysis?
1 reply →
A person would lose their mind pretty quickly this way.
Great potential job for an LLM to potentially spot abuse. The existing mod tools/system seem to work quite well.. I like HN how it is!
Lol. Imagine them having a whole room of tv's running it...
Could be a cool movie scene where the stream of data is actually live HN comments lol.
Another great use of the HN API! It would be nice to filter it somehow to threads where I have commented, sometimes I don't notice until days later that someone replied to me.
To shill my own use of the API:
I did an animated "replay" view for historical threads like Rust 1.0 launch: https://hn.unlurker.com/replay?item=9551937
And a (static, refreshable) view of recent activity grouped by topic: https://hn.unlurker.com
You know about https://www.hnreplies.com/ ? Sends you a mail when someone replies to your comment.
Didn't know about this! Thanks, I just wish something like this was on signal or discord but maybe that's because I don't operate much mail and I really like signal...
But anyways, just as a headsup for a moment I was confused that I wasn't getting the verification mail and I was literally going to comment it but then I got the mail just in time, so maybe to anyone out there, maybe be a bit patient as its worth it (or it could totally just be an issue from my side as I spammed it twice or maybe it got congested I am not sure)
EDIT: It wasn't that they sent me 2 mails but rather that they sent me 1 mail and I clicked on it and then got another and I thought it was because of me spamming verification twice but it turns out that they sent me a mail because someone had responded to me :)
Neato. I really like this.
1 reply →
This has been very reliable for me in the past, great project.
perfect, thanks
Bets on how overwhelmingly negative the comment sentiments are? Anyone got an LLM handy to analyze that
a very very quick copy/paste of 73 comments there - fed to claude with a basic "characterize+count comments" give me 70% neutral/technical, 14% positive, 16% negative.
claude also said "The negative comments tend to be more substantive critiques of systems or research approaches rather than personal attacks." - so... that's good? :-)
I've done it before on large threads. Neutral sentiment tends to dominate at roughly 50% but negative sentiment is much higher than positive sentiment. Negative sentiment is also higher if you just take the highest ranked comments.
It's actually quite stark looking at it this way and makes me question how healthy it is to use this site as often as I do. Fully aware that we're also piling on the negative sentiment.
Okay, fine, let's inject some positivity: OMG! Yay, that's awesome and amazing! That's so wonderful and beautiful!!! Thank You! Kittens! Puppies! Rainbows! Unicorns!
Of course, the next thing is to watch for edits and deletes. What gets written and then gets thought better of. What does raw and unfiltered HN think?
Nothing actually gets deleted on HN, you can turn on "showdead" in your options to see flagged and dead comments. None of that gets sent with the API, though.
There is an API endpoint called "updates" but it's incredibly vague about what it actually shows, or how persistent it is. It might show edits.
if the user deletes their comment, its contents isn't saved
2 replies →
Pretty cool! I must manually refresh to see new posts. Implementing real-time updates (e.x. via WebSocket or Server-Sent Events) would significantly improve usability.
It sounds cool, but for usability its not great. Think about how reddit recalculates the results order as you're paging through, and you see items from the previous page show up on the next. Now imagine that happening in realtime. Maybe there's a link you want to read, but you get pulled away for 10 minutes. By the time you get back the link is higher or lower, or may be completely missing.
As a side feature, that would be neat though.
That is a good point, however there are design solutions around it. For example: - Poll every X seconds instead of real-time - Enable user to toggle real-time mode - Load new posts in with a "+" button at the top to fetch the latest posts (like Twitter)
2 replies →
Writing this comment just to see how quickly it shows up
Edit: Felt like it took almost a minute, I wasn't timing precisely because I didn't expect it to take so long!
It (hopefully) took exactly 30 seconds, the page delays every item until 30 seconds after its posted date. It doesn't poll HN's server, it opens a websocket to the official HN Firebase, and without the delay, items appear in large chunks. I'm pretty sure the HN server syncs with Firebase every 30 seconds, so this is as fast as it can go while still being accurate.
Interestingly some comments appear as [delayed].
This is so cool. Waiting to see my own comment
Saw it!
test
Fun! A css animation would make the updates less jarring and the feed easier to read
It works, but I'd advise about using the same branding. Consider making it significantly different than the website's comments page to avoid any issues, since it's not part of the webapp nor sanctioned.
Ignore this guy, lol.
Okay, maybe you could explain why copying the look of a product to look exactly like the said product, without it being warranted is a worth ignoring and funny?
1 reply →
This is pretty sweet
This is awesome!
[dead]