Crawlspace – A centralized web crawling platform built on Cloudflare
Crawlspace is a centralized web crawling platform that benefits crawler developers AND website owners. Developers can affordably crawl tens of millions of pages per month, scrape with LLMs, and save data in attached storage. Website owners are shielded by a platform-wide TTL cache that absorbs redundant bot traffic. AI bots are running rampant on the open web. Many recent HN stories[1][2][3][4] describe how web crawlers have run amok and hammer websites with DDoS-like traffic. They often do this with blatant disregard of website owners' wishes (e.g. ignoring robots.txt, 429s, Retry-After…
Crawlspace – A centralized web crawling platform built on Cloudflare launched on January 21, 2025 with 9 votes, #2,006 of 6,142 launches that month and more than 75% of that year's launches. The site was still online in September 2026, 1 year on.
What it does
Crawlspace is a centralized platform for developers to build and deploy web crawlers. Gather fresh data for your apps and agents while contributing to a platform-wide cache for crawler traffic.
In the maker’s words, at launch
Crawlspace is a centralized web crawling platform that benefits crawler developers AND website owners. Developers can affordably crawl tens of millions of pages per month, scrape with LLMs, and save data in attached storage. Website owners are shielded by a platform-wide TTL cache that absorbs redundant bot traffic.…
Does a similar job
all alternatives →
Crawlee – Web scraping and browser automation library for Node.js2022 · crawlee.dev · ▲282Hey HN, This is Jan, founder of Apify, a web scraping and automation platform. Drawing on our team's years of experience, today we're launching Crawlee [1], the web scraping and browser automation library for Node.js that's designed for the fastest development and maximum reliability in production. For details, see the short video [2] or read the announcement blog post [3]. Main features: - Supports headless browsers with Playwright or Puppeteer - Supports raw HTTP crawling with Cheerio or JSDOM - Automated parallelization and scaling of crawlers for best performance - Avoids blocking using…
Crawlab: Open-Source Web Crawler Admin Platform That Runs Any Language2019 · github.com · ▲116Distributed web crawler admin platform for spiders management regardless of languages and frameworks. - tikazyq/crawlab
- AMA Multi-billion page web crawler in the cloud2017 · mixnode.com · ▲14
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 21d ago · stillwet.art
PixVerse R2▲381A real-time world model you can explore and change
AI · 30d ago · world.pixverse.video
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, January 2025
the whole month →
Hello! I'm Byran. I spent the past ~6 months engineering a laptop from scratch. It's fully open-source on GH at: https://github.com/Hello9999901/laptop
Dev tools · 2025 · byran.ee
- TITetris in a PDF▲1,289
I realized that the PDF engines of modern desktop browsers (PDFium and PDF.js) support JavaScript with enough I/O primitives to make a basic game like Tetris. It was a bit tricky to find a union of features that work in both engines, but in the end it turns out that showing/hiding annotation "fields" works well to make monochrome pixels, and keyboard input can be achieved by typing in a text input box. All in all it's quite janky but a nice reminder of how general purpose PDF scripting can be. The linked PDF is all ASCII so you can just open it in a text editor, or have a look at…
Life & fun · 2025 · th0mas.nl
21st.dev▲1,211I originally built this app to organize and reuse interesting React + Tailwind components I came across on Twitter, inspired by the shadcn/ui approach. Later, a friend started contributing their own components, and it got me thinking—this could grow into something bigger, like an NPM for design engineers to share and use components. Would love to hear your thoughts!
Dev tools · 2025 · 21st.dev
Create lifelike, personalized AI avatars from text prompts
AI · 2025 · jogg.ai
CSS library based on Counter Strike 1.6 UI.
Life & fun · 2025 · cs16.samke.me