Dev tools · alternatives · 2026
24 alternatives to Crawlab: Open-Source Web Crawler Admin Platform That Runs Any Language
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. - tikazyq/crawlab
Crawlab is an open-source web crawler management platform that enables developers to build and run crawlers in any programming language. It provides a centralized interface for creating, scheduling, and monitoring web… Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes. Crawlab: Open-Source Web Crawler Admin Platform That Runs Any Language launched in 2019; newer entries below may have overtaken it.
- 1
- 2
Crawlab▲1Web crawler admin platform for any framework or language
2020 · github.com · its alternatives →
- 3
- 4
Hey HN, This is Jan, founder of Apify, a web scraping and automation platform. Drawing on our team's years of experience, today we're launching Crawlee [1], the web scraping and browser automation library for Node.js that's designed for the fastest development and maximum reliability in production. For details, see the short video [2] or read the announcement blog post [3]. Main features: - Supports headless browsers with Playwright or Puppeteer - Supports raw HTTP crawling with Cheerio or JSDOM - Automated parallelization and scaling of crawlers for best performance - Avoids blocking using…
2022 · crawlee.dev · its alternatives →
- 5
Distributed crawler powered by Headless Chrome. Contribute to yujiosaka/headless-chrome-crawler development by creating an account on GitHub.
2018 · github.com · its alternatives →
- 6
A Powerful Spider(Web Crawler) System in Python. Contribute to binux/pyspider development by creating an account on GitHub.
2014 · github.com · its alternatives →
- 7
CrawlRaven▲287SEO Hub for GSC + GA4 + a 200-point crawl
Jul 2026 · crawlraven.com · its alternatives →
- 8
Crawler.sh▲349Ultra-fast desktop & CLI app that finds your SEO & AEO issues and allows you to extract your content as Markdown
Mar 2026 · crawler.sh · its alternatives →
- 9
Spider Free▲354Turn websites into organized data without code.
2019 · chromewebstore.google.com · its alternatives →
- 10
Learn how to scrape webpages using Puppeteer and Serverless Functions built with OpenFaaS.
2020 · openfaas.com · its alternatives →
- 11
- 12
Please try out this website analyzer and provide feedback on how to improve the output report so that you love the tool. In the first few sentences of the README you will also find a link to the GUI application if you don't want the CLI. To give you an idea of the report, I am sending a sample HTML report for Apple.com - https://crawler.siteone.io/html/2024-08-19/forever/v-bnb7tu5...
2024 · github.com · its alternatives →
- 13
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headf
2018 · github.com · its alternatives →
- 14
Minimalist and powerful Web Crawler. Contribute to untwisted/sukhoi development by creating an account on GitHub.
2017 · github.com · its alternatives →
- 15DR
I'd like to invite everyone to try out DontBeEvil.rip, an experimental search engine for developers. tl;dr $ alias rip="curl -G -H 'Accept: text/plain' --url https://dontbeevil.rip/search --data-urlencode " $ rip 'q=Heartbleed bug' DontBeEvil.rip is a year long experiment to see if a small team can build a developer-focused search engine that is self-sustaining on $10 monthly subscriptions. It works by only indexing high-quality resources that are relevant to developers. You won't get useless listicles because we'll never crawl them. Relevant urls are harvested from HN,…
2022 · its alternatives →
- 16AT
2021 · iwishtherewas.app · its alternatives →
- 17SE
2017 · openstar.pw · its alternatives →
- 18
Cross-platform persistent and distributed web crawler :link: - schollz/linkcrawler
2017 · github.com · its alternatives →
- 19SA
Sitebulb is the revolutionary website crawler for better SEO audits. Twice winner of Best Search Software Tool. Desktop & Cloud crawling without compromise!
2017 · sitebulb.com · its alternatives →
- 20
WebAutomation.io▲154The biggest marketplace of readymade no code web scrapers
2021 · webautomation.io · its alternatives →
- 21
- 22
Crawlspace is a centralized web crawling platform that benefits crawler developers AND website owners. Developers can affordably crawl tens of millions of pages per month, scrape with LLMs, and save data in attached storage. Website owners are shielded by a platform-wide TTL cache that absorbs redundant bot traffic. AI bots are running rampant on the open web. Many recent HN stories[1][2][3][4] describe how web crawlers have run amok and hammer websites with DDoS-like traffic. They often do this with blatant disregard of website owners' wishes (e.g. ignoring robots.txt, 429s, Retry-After…
2025 · crawlspace.dev · its alternatives →
- 23
- 24
Also compare
Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →