← Back to context

Comment by dredmorbius

2 months ago

For one datapoint ...

I have a custom HN CSS which includes some formatting of different sets of user accounts. Admins, for example, get orange highlighting and a dragon emoji (for one does not meddle in the affairs of ...).

Also included are leaders, which is the one part of my CSS build script which is, or at least was until a few minutes ago, dynamic. Presently HN is returning "sorry" to my curl request. Given that I run that build manually a few times a month, it's not a matter of hitting HN with frequent scrapes. But HN has become increasingly scrape-hostile over time.

Back in 2023 I did a crawl of all of HN's front-page daily history (365.25 days/year * 17 years, so about 6,200 requests), to answer a question which had come up about what was/wasn't mentioned in submission titles. That scrape included a delay (probably either 1 or 10 seconds, possibly more, I don't recall which and may have run the fetch directly from the command line), and ran (initially) without issues. I don't think it would fly today.

I reported on findings at the time and several times since:

<https://hn.algolia.com/?dateRange=all&page=0&prefix=false&qu...>

8 comments

dredmorbius

Reply
fragmede  2 months ago

HN is exported to firebase, which you can hit directly, for that sort of purpose

https://github.com/HackerNews/API

  • dredmorbius  2 months ago

    I know that.

    I've not worked with the API, and there's the blessing/curse (blurse‽) that HTML is a known, if poor, standard.

    API always translates to "one more thing to learn, that's applicable to a single-use case". HTML scraping / sorting I can apply across multiple sites.

    That said, a standard, say, JSON packaging of website contents available on request might be fun to have.

  • dredmorbius  2 months ago

    Looking at the API ...

    ... it's starting to make sense, but ...

    ... the API is geared at requesting specific content items (posts, comments, users). There doesn't seem to be a way to directly make a request for a front-page history page (that is, the 30 items archived on a given date. Say, 2008-11-05:

    <https://news.ycombinator.com/item?id=29769470>

    I could look more into their methodology to see if I can use similar approaches.

    The existence of "dead" and "deleted" values does seem interesting. I might do some playing with those to see what shows up (I suspect that most additional information is suppressed...)

    OK, looking at a recent dead atomic128 comment:

      $ curl -s 'https://hacker-news.firebaseio.com/v0/item/48820709.json?print=pretty'
  {
    "by" : "atomic128",
    "dead" : true,
    "id" : 48820709,
    "parent" : 48819517,
    "text" : "[flagged]",
    "time" : 1783444517,
    "type" : "comment"
  }

    So userID is visible.

    And from a current dead submission in the New queue:

      $ curl -s 'https://hacker-news.firebaseio.com/v0/item/48868688.json?print=pretty'
  {
    "by" : "millwright-sw",
    "dead" : true,
    "id" : 48868688,
    "score" : 1,
    "time" : 1783743361,
    "type" : "story"
  }

    That's missing the title and URL, as I suspected it would, though the submitter UID is available.

    To get top stories by date I'd actually have to submit more requests, walking through item numbers, splitting out comments and stories. Based on Whaly's 2021 retrospective, with about 4.2 million items (stories + comments) posted in total, that's about 12,000 items per day. Versus, well, one "Past" page result...