Alternatives

Products that do what Partial content web crawling using HTTP/2 and Go does

Hi, I wrote a low-level HTTP/2 web crawler in Go, which can scrape partial content to save traffic. Tl;dr e.g. the HTML of a YouTube video contains the video description, views, likes etc. in its first 600KB, the remaining 900KB are of no use for me, but I have to pay my proxies by the gigabyte. My crawler receives packet per packet, and if I got everything I needed I reset the request, and only pay-for-what-i-crawled. This is also potentially useful for large-scale crawling operations, where duplicates matter. You could compute a simHash on the fly, and reset on-the-fly before crawling…

1. 1
   Crawlee▲229

   Build reliable web scrapers and robots, fast!

   2022

2. 2
   Web Scraper API▲287

   Scrape the web at scale without getting blocked

   2021

3. 3
   Proxy Tester by ScrapeOps▲208

   Benchmark proxies for reliable, target-specific scraping

   28d ago · scrapeops.io

4. 4
   Cloudflare Pay Per Crawl▲167

   No AI crawl without compensation! ✊

   2025

5. 5
   Web Scraper API▲142

   Scrape the web at scale without getting blocked

   2023

6. 6
   Trace▲156

   F**k The Algorithm. Make Your Own

   Jan 2026

7. 7AG
   A Go web server with logging, tracing, and demo apps in a single file▲116

   2019 · github.com

8. 8ST
   Scrape the web by injecting JavaScript into web pages▲109

   2014 · espion.io

9. 9
   Geekflare Scraping API v2▲84

   RAG-ready web scraping that cuts your LLM token costs

   Apr 2026

10. 10
    LLM SEO Index Crawler Check▲157

    Check if your website can get crawled by ChatGPT

    2025

11. 11
    Crawly▲80

    Uptime, SSL certificate, missing assets, status pages

    2020

12. 12
    ScrapeAll▲87

    Automated web data scraper

    2022

13. 13AD
    A distributed and persistent web crawler written in Go▲5

    2017 · github.com

14. 14GS
    GoPee – Simple, Self-Hosted Web Proxy in Go▲70

    2015 · gopee.herokuapp.com

15. 15IW
    I wrote a tiny Python-based HN crawler with scrapy▲53

    2012 · github.com

16. 16
    Proxy API for Web Scraping▲84

    Get HTML from any page with a simple API call

    2023

17. 17H2
    HTTP/2 HPACK Compression Go Library▲40

    2016 · github.com

18. 18GB
    Getsy – browser/client-side web scraper▲49

    2017 · github.com

19. 19CA
    crawl a website and store it in S3 from your browser▲43

    2011 · github.com

20. 20IH
    I have written a cloud native dark web crawler in Go▲7

    2021 · github.com

21. 21P3
    Python 3.5 Async Web Crawler Example▲28

    2016 · github.com

22. 22IC
    I created a tiny web crawler for Python▲5

    2024 · github.com

23. 23MS
    My side project that searches the web for technologies▲22

    Hello HN! I've been working on http://underthesite.com for the last month and now think it is ready for some full strength HN feedback. What do you guys think? It crawls up to 10 pages of a given site while you wait, looking for community-provided CSS / XPath selectors and regular expressions. Additionally, I'd like to appeal to you to submit matchers for technologies that you care about. Technologies are easy to add, so add your favorite jQuery plugins, analytics tools, client-side node.js wrappers, what have you. I'm going to be running a large crawl in the next few days and want to make…

    2011

24. 24AM
    A Multi-billion page web crawler in the cloud▲14

    2017 · mixnode.com

Ranked by how close each launch is in meaning, then by votes. Refine with a description →