Alternatives

Products that do what A Multi-billion page web crawler in the cloud does

1. 1OS
   Open-source search engine with 2bn-page index▲229

   2016 · deusu.org

2. 2CO
   Crawlab: Open-Source Web Crawler Admin Platform That Runs Any Language▲116

   2019 · github.com

3. 3AE
   An experimental, people-powered search engine▲215

   2021 · ninfex.com

4. 4CC
   CloudScrape – Cloud-based web scraping platform▲116

   2015 · cloudscrape.com

5. 5AD
   A distributed and persistent web crawler written in Go▲5

   2017 · github.com

6. 6AH
   Apifier – hosted web crawler for developers▲101

   2015 · apifier.com

7. 7AO
   An open source web crawler for the Mwmbl non-profit search engine▲6

   2022 · github.com

8. 8SA
   Scraperjs – A versatile web scraper▲192

   2014 · github.com

9. 9AM
   A modular, durable web-crawler for Clojure▲29

   2016 · github.com

10. 10AS
    Apify SDK – A scalable web crawling and scraping library for JavaScript▲78

    2018 · github.com

11. 11SA
    Sitebulb, a website crawler and auditor for SEOs▲120

    2017 · sitebulb.com

12. 12AO
    An open distributed search engine for science▲98

    2014 · juretriglav.si

13. 13CB
    Cloudstitch – Build and share spreadsheet-powered web content▲84

    2015 · cloudstitch.io

14. 14TL
    Torrentinim – low memory-footprint, API-only torrent search engine▲117

    2021 · github.com

15. 15SM
    (Sukhoi) Minimalist and Powerful Web Crawler in Python▲6

    2017 · github.com

16. 16SA
    Stork: A customizable, WASM-powered full-text search plugin for the web▲137

    2020 · stork-search.net

17. 17TD
    The dht(distributed hash table) network crawler▲5

    2015 · github.com

18. 18SA
    Sukhoi A highly flexible Web Crawler in Python▲5

    2020 · github.com

19. 19HC
    Highly cost-efficient search engine in Rust▲61

    2021 · github.com

20. 20AW
    A web crawler that builds word frequency lists for websites▲9

    2018 · github.com

21. 21IH
    I have written a cloud native dark web crawler in Go▲7

    2021 · github.com

22. 22PD
    Presearch: Decentralized Search Engine▲10

    2021 · presearch.org

23. 23LT
    Lightweight Torrent Search Engine▲13

    2014 · nexxus.io

24. 24DT
    Diggernaut – Turn website content into data sets▲49

    2016 · diggernaut.com

Ranked by how close each launch is in meaning, then by votes. Refine with a description →