Partial content web crawling using HTTP/2 and Go
Hi, I wrote a low-level HTTP/2 web crawler in Go, which can scrape partial content to save traffic. Tl;dr e.g. the HTML of a YouTube video contains the video description, views, likes etc. in its first 600KB, the remaining 900KB are of no use for me, but I have to pay my proxies by the gigabyte. My crawler receives packet per packet, and if I got everything I needed I reset the request, and only pay-for-what-i-crawled. This is also potentially useful for large-scale crawling operations, where duplicates matter. You could compute a simHash on the fly, and reset on-the-fly before crawling…
What it does
In the maker’s words, at launch
Hi, I wrote a low-level HTTP/2 web crawler in Go, which can scrape partial content to save traffic. Tl;dr e.g. the HTML of a YouTube video contains the video description, views, likes etc. in its first 600KB, the remaining 900KB are of no use for me, but I have to pay my proxies by the gigabyte. My crawler receives packet per packet, and if I got everything I needed I reset the request, and only pay-for-what-i-crawled. This is also potentially useful for large-scale crawling operations, where duplicates matter. You could compute a simHash on the fly, and reset on-the-fly before crawling the entire document (again).
Does the same job
all alternatives →
Proxy Tester by ScrapeOps28d ago · scrapeops.io · ▲208Benchmark proxies for reliable, target-specific scraping
- AGA Go web server with logging, tracing, and demo apps in a single file2019 · github.com · ▲116
More growth this month
the category →
AstraPixels▲267A pixel-art solar system at its real current positions.
Growth · 29d ago · astrapixels.com
Launched alongside, January 2026
the whole month →- IN
Hey HN! I wanted to share something I built over the last few weeks: isometric.nyc is a massive isometric pixel art map of NYC, built with nano banana and coding agents. I didn't write a single line of code. Of course no-code doesn't mean no-engineering. This project took a lot more manual labor than I'd hoped! I wrote a deep dive on the workflow and some thoughts about the future of AI coding and creativity: http://cannoneyed.com/projects/isometric-nyc
AI · Jan 2026 · cannoneyed.com
Automatic AI-powered code reviews the moment you open a PR
Dev tools · Jan 2026 · kilo.ai
