Alternatives

Products that do what I wrote a tiny Python-based HN crawler with scrapy does

1. 1IC
   I created a tiny web crawler for Python▲5

   2024 · github.com

2. 2VW
   Voyager – write a web crawler/scraper as a state machine in Rust▲110

   2020 · github.com

3. 3SM
   (Sukhoi) Minimalist and Powerful Web Crawler in Python▲6

   2017 · github.com

4. 4SA
   Scraperjs – A versatile web scraper▲192

   2014 · github.com

5. 5CO
   Crawlab: Open-Source Web Crawler Admin Platform That Runs Any Language▲116

   2019 · github.com

6. 6AM
   A modular, durable web-crawler for Clojure▲29

   2016 · github.com

7. 7IW
   I wrote an Enigma machine emulator for the Web▲83

   2019 · enigma.lh.mk

8. 8SE
   Search Engine and Crawler in Python▲21

   2017 · openstar.pw

9. 9AP
   A pure Python implemention of Apache Thrift▲64

   2014 · thriftpy.readthedocs.org

10. 10SA
    Sukhoi – A flexible and extensible Webcrawler in Python▲131

    2017 · github.com

11. 11GB
    Getsy – browser/client-side web scraper▲49

    2017 · github.com

12. 12AH
    Apifier – hosted web crawler for developers▲101

    2015 · apifier.com

13. 13IB
    I built Multy – Generate a short URL to share a list of websites▲128

    2021 · multy.me

14. 14SA
    Sukhoi A highly flexible Web Crawler in Python▲5

    2020 · github.com

15. 15ID
    I developed a no code web scraper for effortless data extraction▲5

    2025 · scrapementor.com

16. 16IH
    I have written a cloud native dark web crawler in Go▲7

    2021 · github.com

17. 17AS
    Apify SDK – A scalable web crawling and scraping library for JavaScript▲78

    2018 · github.com

18. 18AD
    A distributed and persistent web crawler written in Go▲5

    2017 · github.com

19. 19IB
    I built a free oEmbed API for the web▲61

    2013 · oembed.io

20. 201D
    (1 day project) I crawled +50k subreddits and made an interactive graph▲14

    2018 · github.com

21. 21FC
    Fitter CLI – powerful command line scrapper with browser support▲7

    2023 · github.com

22. 22CS
    Cocktail search written in Python using scrapy and sphinx▲29

    2012 · cocktails.p24dev.de

23. 23DS
    Distributed Scraper▲27

    2017 · stdlib.com

24. 24AM
    A Multi-billion page web crawler in the cloud▲14

    2017 · mixnode.com

Ranked by how close each launch is in meaning, then by votes. Refine with a description →