Crawlee – Web scraping and browser automation library for Node.js

In plain words

Crawlee is a Node.js library for web scraping and browser automation that supports both headless browsers like Playwright and Puppeteer, as well as raw HTTP crawling with Cheerio or JSDOM. It handles parallelization and scaling automatically, manages URL queues, and includes features to avoid blocking such as smart sessions and proxy rotation. Built entirely in TypeScript, it offers comprehensive documentation and tutorials. The library is designed for developers who need fast development cycles and reliable production performance.

written from the facts on this page · September 2026

From the sources

After months of development, polishing, and community feedback, Crawlee for Python is leaving beta and entering a production/stable development status. We are happy to announce Crawlee for Python v1.0.0. Check out all the new features and updates now!

In the maker’s words, at launch

Hey HN, This is Jan, founder of Apify, a web scraping and automation platform. Drawing on our team's years of experience, today we're launching Crawlee [1], the web scraping and browser automation library for Node.js that's designed for the fastest development and maximum reliability in production. For details, see…

• 
  Apify SDK – A scalable web crawling and scraping library for JavaScript2018 · github.com · ▲78

  Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headf

• 
  Crawlee2022 · crawlee.dev · ▲229

  Build reliable web scrapers and robots, fast!

• 
  Crawly2016 · crawly.diffbot.com · ▲606

  Never write another web scraper

• 
  Headless Chrome Crawler2018 · github.com · ▲172

  Distributed crawler powered by Headless Chrome. Contribute to yujiosaka/headless-chrome-crawler development by creating an account on GitHub.

• 
  scrape-it2016 · github.com · ▲186

  A Node.js scraper for humans.

• 
  Web scraping that just works with OpenFaaS with Puppeteer2020 · openfaas.com · ▲145

  Learn how to scrape webpages using Puppeteer and Serverless Functions built with OpenFaaS.

• 
  /dev/null▲468

  a social network for devs

  Dev tools · 2d ago · dev-null.sh

• 
  Whiteboard (YC W26) – an open-source IDE for thoughtful software design▲423

  open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.

  Dev tools · 27d ago · github.com

• 
  Reladraw – A diagram language where you decide where to place things▲411

  Contribute to reladraw/reladraw development by creating an account on GitHub.

  Dev tools · 25d ago · github.com

• 
  OpenBot▲362

  Grok Bot alternative: free, local, open-source, multiplayer

  Dev tools · 22d ago · openbot.run

• 
  Talanoa▲340

  Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…

  Dev tools · 20d ago · talanoa.email

• 
  CoreSpeed▲332

  One MCP for everything your agents need: apps, memory, tools

  Dev tools · 27d ago · corespeed.io

• 
  Xnapper▲1,517

  Take beautiful screenshots instantly

  Work · 2022 · xnapper.com

• 
  Gamma▲1,169

  Auto-generate decks, docs & social posts inside any workflow

  Work · 2022 · gamma.app

• 
  Threado▲921

  The command center for community builders

  Life & fun · 2022

• 
  Remotebase▲744

  Build remote engineering teams in 24 hours

  Dev tools · 2022 · remotebase.com

• 
  DhiWise 2.0▲740

  Convert Figma designs into Flutter 3.0 & React code rapidly

  Dev tools · 2022 · dhiwise.com

• PA
  Pornpen.ai – AI-Generated Porn▲711

  Hey HN, I've been working on https://pornpen.ai, a site for generating adult images. Please only visit the site if you are 18+ and willing to look at NSFW images. This site is an experiment using newer text-to-image models. I explicitly removed the ability to specify custom text to avoid harmful imagery from being generated. New tags will be added once the prompt-engineering algorithm is fine-tuned further. If the servers are overloaded, take a look at the feed and search pages to look through past results. For comments/suggestions/feedback please visit…

  AI · 2022 · pornpen.ai