← Back to context

Comment by sakai

13 years ago

I'll consider that a challenge...

To any that have experience getting comments data from HN -- what's the fastest, most polite way to do this? And am I correct in remembering that there's some aggressive rate-limiting for crawling the site?

3 comments

sakai

Reply
pg  13 years ago

Don't crawl the site, please. The place to get data is the HNSearch API.

malandrew  13 years ago

Any plans to try to get a dataset for supervised ml? Perhaps collect the top 4 comments from all front page threads and post a survey on HN asking HNers to rate those comments for skepticism/dismissiveness?