Exploring HN by mapping and analyzing 40M posts and comments for fun
Accessible end-to-end deep dive of crawling, embedding, projecting, and semantic searching all of Hacker News for my Hackerverse project.
Exploring HN by mapping and analyzing 40M posts and comments for fun launched on May 9, 2024 with 520 votes, #51 of 4,539 launches that month and more than 99% of that year's launches. The site was still online in September 2026, 2 years on.
In plain words
Exploring HN is a project that maps and analyzes 40 million posts and comments from Hacker News. It lets users discover patterns, trends, and insights across the platform's discussion history. The tool is designed for people curious about online communities and data exploration. It stands out by processing a massive dataset to reveal how conversations and topics have evolved on one of the web's most influential tech discussion forums.
written from the facts on this page · September 2026
From the sources
The above is a map of all Hacker News posts since its founding , laid semantically i.e. where there should be some relationship between positions and distances. I've been building it and some other interesting stuff over the past few weeks, to play around with text embeddings . Given that HN has a lot of interesting, curated content and exposes all its content programatically , I thought it'd be a fun place to start. A quick primer of embeddings: they are a powerful and cool way to represent something (in this case, text) as a point in a high-dimensional space , which in practical terms just means an array of floats, one for its coordinate in that dimension. The absolute position doesn't…from blog.wilsonl.in
Does a similar job
all alternatives →- ATAnalyzing top HN posts with language models2022 · ▲117
Hi HN, I spent a few weeks looking at the top HN posts of all time. This included exploration, clustering, creating visualizations, and zooming in on what (to me personally) seems like some of the best discussions on here. Three things in this post: 1- The interesting groups of HN posts 2- The interactive visualizations that you can explore in your browser 3- The data from this exploration -- this includes CSV of the titles as well as the text embeddings of 3,000 Ask HN articles. Blog post about this whole process here: [1] ============ 1- The interesting groups of HN posts From the…
- SHSearch HN for interesting comment sections2024 · searchhacker.news · ▲60
I built this tool to help me find interesting discussions on Hacker News. I love reading HN discussions almost more than the articles themselves. However, I found that full text search, although highly performant, is not always good at surfacing interesting discussions on a certain topic -- especially if you don't know what to search for exactly. I built this by scraping the most recent ~6 million posts (that's about 2 years of history) and putting the resulting posts and their vector embeddings into Postgres. Let me know what could be improved, and if you'd like a more detailed writeup of…
- ADA discovery-focused search engine for Hacker News2024 · hn.trieve.ai · ▲193
We (Nick, Dens, Denzell, Fede, Drew, Aaryan, and Daniel) have been building HN Discovery, a discovery-focused search engine for Hacker News, in our spare time for the past 6 months and are excited to show it! It adds the following features relative to the existing keyword search interface and preserves the existing ones: - no-JS version (hnnojs.trieve.ai) - site:{required_site} and site:{negated-site} filters - public analytics - LLM generated query suggestions based on random stories - recommendations - dense vector semantic search - SPLADE fulltext search - RAG AI chat - order by…
40k HN comments mentioning books, extracted using deep learning2021 · hacker-recommended-books.vercel.app · ▲1,359Thinking, Fast and Slow - with 523 comments from the Hacker News community
- MHMeet.hn – Meet the Hacker News community in your city2024 · ▲768
Hey HN! I just published https://meet.hn, a map to find hackers in your city. How it works? Demo of the signup process: https://x.com/meet_hn/status/1834918518904746329 1. Fill the form: username, city+country 2. Copy the text generated in the box below the form, and paste it in your HN description. 3. Click "Add me on the map" Optionnaly (it's recommended!) you can add links to your socials as well as some tags to showcase your interests. Why does it exist? I created this because, despite its harsh reputation on the internet, I love the HN community. I…
Hacker Search – A semantic search engine for Hacker News2024 · hackersearch.net · ▲233Hi HN! I'm Jonathan and I built Hacker Search (https://hackersearch.net), a semantic search engine for Hacker News. Type a keyword or a description of what you're interested in, and you'll get top links from HN surfaced to you along with brief summaries. Unlike HN's otherwise very valuable search feature, Hacker Search doesn't require you to get your keywords exactly right. That's achieved by leveraging OpenAI's latest embedding models alongside more traditional indexes extracted from the scraped and cleaned up contents of the links. I think there are many more interesting things…
More work this month
the category →
Endless lofi beats composed live in your browser, over pixel-art nights in Paris, Tokyo, New York, Hong Kong, Sydney, San Francisco and more. Free, no sign-up.
Work · 25d ago · loficities.com
Smooth Recorder▲257Mac screen recordings that look edited when you hit stop
Work · 19d ago · smoothrecorder.com
Work · 13d ago · readrare.com
Source: Show HN post ↗
