Crawlab: Open-Source Web Crawler Admin Platform That Runs Any Language
Does the same job
all alternatives →- WWWarcDB: Web crawl data as SQLite databases2022 · github.com · ▲171
- AHApifier – hosted web crawler for developers2015 · apifier.com · ▲101
- CWCrawlee – Web scraping and browser automation library for Node.js2022 · crawlee.dev · ▲282
Hey HN, This is Jan, founder of Apify, a web scraping and automation platform. Drawing on our team's years of experience, today we're launching Crawlee [1], the web scraping and browser automation library for Node.js that's designed for the fastest development and maximum reliability in production. For details, see the short video [2] or read the announcement blog post [3]. Main features: - Supports headless browsers with Playwright or Puppeteer - Supports raw HTTP crawling with Cheerio or JSDOM - Automated parallelization and scaling of crawlers for best performance - Avoids blocking using…
- GAGutenberg – A static site engine as a single executable2017 · getgutenberg.io · ▲241
- SASitebulb, a website crawler and auditor for SEOs2017 · sitebulb.com · ▲120
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
Lettertrace▲375Track your AI visibility for free (using your own API keys!)
Dev tools · 25d ago · lettertrace.com
Launched alongside, August 2019
the whole month →
Startup Decks▲711View 100s of startup pitch decks or submit your own.
Growth · 2019 · startupdecks.co
- CT
Dev tools · 2019 · github.com
