Crawlab: Open-Source Web Crawler Admin Platform That Runs Any Language

In plain words

Crawlab is an open-source web crawler management platform that enables developers to build and run crawlers in any programming language. It provides a centralized interface for creating, scheduling, and monitoring web scraping tasks across multiple projects. The platform is designed for developers who need to manage complex crawling operations without being limited to a single language, offering flexibility in implementation while simplifying administration and execution at scale.

written from the facts on this page · September 2026

• 
  Crawlee2022 · crawlee.dev · ▲229

  Build reliable web scrapers and robots, fast!

• 
  Crawlab2020 · github.com · ▲1

  Web crawler admin platform for any framework or language

• 
  Crawly2016 · crawly.diffbot.com · ▲606

  Never write another web scraper

• 
  Crawlee – Web scraping and browser automation library for Node.js2022 · crawlee.dev · ▲282

  Hey HN, This is Jan, founder of Apify, a web scraping and automation platform. Drawing on our team's years of experience, today we're launching Crawlee [1], the web scraping and browser automation library for Node.js that's designed for the fastest development and maximum reliability in production. For details, see the short video [2] or read the announcement blog post [3]. Main features: - Supports headless browsers with Playwright or Puppeteer - Supports raw HTTP crawling with Cheerio or JSDOM - Automated parallelization and scaling of crawlers for best performance - Avoids blocking using…

• 
  Headless Chrome Crawler2018 · github.com · ▲172

  Distributed crawler powered by Headless Chrome. Contribute to yujiosaka/headless-chrome-crawler development by creating an account on GitHub.

• 
  A Python Spider System with Web UI2014 · github.com · ▲159

  A Powerful Spider(Web Crawler) System in Python. Contribute to binux/pyspider development by creating an account on GitHub.

• 
  /dev/null▲483

  a social network for devs

  Dev tools · 3d ago · dev-null.sh

• 
  Whiteboard (YC W26) – an open-source IDE for thoughtful software design▲423

  open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.

  Dev tools · 27d ago · github.com

• 
  Reladraw – A diagram language where you decide where to place things▲411

  Contribute to reladraw/reladraw development by creating an account on GitHub.

  Dev tools · 25d ago · github.com

• 
  OpenBot▲362

  Grok Bot alternative: free, local, open-source, multiplayer

  Dev tools · 23d ago · openbot.run

• 
  Talanoa▲345

  Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…

  Dev tools · 20d ago · talanoa.email

• 
  CoreSpeed▲332

  One MCP for everything your agents need: apps, memory, tools

  Dev tools · 28d ago · corespeed.io

• 
  Startup Decks▲708

  View 100s of startup pitch decks or submit your own.

  Growth · 2019 · startupdecks.co

• 
  Dashblock▲696

  Turn any website into an API

  Dev tools · 2019

• 
  CLI tool for saving web pages as a single file▲640

  ⬛️ CLI tool and library for saving complete web pages as a single HTML file - Y2Z/monolith

  Dev tools · 2019 · github.com

• 
  Get waves▲602

  A simple web app to generate svg waves, unique every time

  Dev tools · 2019 · getwaves.io

• 
  Spider Pro▲574

  The easiest way to scrape the internet.

  Dev tools · 2019

• 
  Marketing Examples▲560

  A gallery of real world marketing examples

  Growth · 2019 · marketingexamples.com