Hacker News front-page analytics
A question about what states were most-frequently represented on the HN homepage had me do some quick querying via Hacker News's Algolia search ... which is NOT limited to the front page. Those results were ... surprising (Maine and Iowa outstrip the more probable results of California and, say, New York). Results are further confounded by other factors.
Thread: https://news.ycombinator.com/item?id=36076870
HN provides an interface to historical front-page stories (https://news.ycombinator.com/front), and that can be crawled by providing a list of corresponding date specifications, e.g.:
https://news.ycombinator.com/front?day=2023-05-25
Easy enough.
So I'm crawling that and compiling a local archive. Rate-limiting and other factors mean that's only about halfway complete, and a full pull will take another day or so.
But I'll be able to look at story titles, sites, submitters, time-based patterns (day of week, day of month, month of year, yearly variations), and other patterns. There's also looking at mean points and comments by various dimensions.
Among surprises are that as of January 2015, among the highest consistently-voted sites is The Guardian. I'd thought HN leaned consistently less liberal.
The full archive will probably be < 1 GB (raw HTML), currently 123 MB on disk.
Contents are the 30 top-voted stories for each day since 20 February 2007.
If anyone has suggestions for other questions to ask of this, fire away.
And, as of early 2015, top state mentions are:
1. new york: 150
2. california: 101
3. texas: 39
4. washington: 38
5. colorado: 15
6. florida: 10
7. georgia: 10
8. kansas: 10
9. north carolina: 9
10. oregon: 9
NY is highly overrepresented (NY Times, NY Post, NY City), likewise Washington (Post, Times, DC). Adding in "Silicon Valley" and a few other toponyms boosts California's score markedly. I've also got some city-based analytics.
> I'd thought HN leaned consistently less liberal.
Why did you think that?
Vanilla liberalism seems to be the most common among the tech crowd (though quieter than exotic political stances.)
@WomanCorn Part of that's a sense from actively using the site (someone else's recent analysis had me as the 16th most active commenter during their analysis period, a bit of a shock to me, honestly), and much from the criticisms you'll find about HN particularly on the Fediverse.
Techbro / Rothbardian/Randian Libertarian culture, especially, are mentioned / prevalent / frequently encountered in my experience.
One reason I'm looking at the domain (submission site) stats is to get a better sense of that. My crawl's just under 2/3 complete, I'm waiting on more recent trends to start diving in deeply, though I'm querying data and testing code as the data come in.