SiteOne Crawler – in-depth website analyzer and exporter (open-source)

Please try out this website analyzer and provide feedback on how to improve the output report so that you love the tool. In the first few sentences of the README you will also find a link to the GUI application if you don't want the CLI. To give you an idea of the report, I am sending a sample HTML report for Apple.com - https://crawler.siteone.io/html/2024-08-19/forever/v-bnb7tu5...

Visit github.com ↗Alternatives

In plain words

SiteOne Crawler is an open-source website analyzer and exporter that scans websites and generates detailed reports. It offers both a command-line interface and a graphical application for users who prefer not to use the CLI. The tool produces comprehensive HTML reports documenting website structure and data, making it useful for developers and website auditors who need to understand site composition and performance metrics.

written from the facts on this page · September 2026

Does the same job

all alternatives →

Crawler.shMar 2026 · ▲352
Free Local AEO & SEO Spider and a Markdown content extractor
Crawly2016 · ▲601
Never write another web scraper
CrawlRavenJul 2026 · crawlraven.com · ▲287
SEO Hub for GSC + GA4 + a 200-point crawl
Sitebulb2017 · ▲146
Crawl and visualize your website structure
Crawlee2022 · ▲229
Build reliable web scrapers and robots, fast!
CW
Crawlee – Web scraping and browser automation library for Node.js2022 · crawlee.dev · ▲282
Hey HN, This is Jan, founder of Apify, a web scraping and automation platform. Drawing on our team's years of experience, today we're launching Crawlee [1], the web scraping and browser automation library for Node.js that's designed for the fastest development and maximum reliability in production. For details, see the short video [2] or read the announcement blog post [3]. Main features: - Supports headless browsers with Playwright or Puppeteer - Supports raw HTTP crawling with Cheerio or JSDOM - Automated parallelization and scaling of crawlers for best performance - Avoids blocking using…

More dev tools this month

the category →

Dograh▲592
The open source VAPI alternative
Dev tools · 25d ago · dograh.com
Meridian▲530
Don't let your work go unnoticed. Get promoted!
Dev tools · 20d ago · meridiona.com
x1▲516
Lovable for iPhone apps go from idea to App Store
Dev tools · 11d ago · x1.new
Computable GPU Index (CGI)▲424
The first open-source price index for GPU compute
Dev tools · 10d ago · getcomputable.com
Open-Source eInk Bike Computer▲402
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 2d ago · opentrailpaper.com
The GTM Co-Founder▲402
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com

Launched alongside, August 2024

the whole month →

Wordware▲9,866
Your tool for building AI agents with natural language
AI · 2024 · wordware.ai
IY
If YouTube had actual channels▲2,741
Life & fun · 2024 · ytch.xyz
Me.bot▲1,628
The inspiring companion for your life
AI · 2024 · me.bot
Upmetrics AI▲1,197
Business plan & financial forecast using AI
Work · 2024 · upmetrics.co
Jupitrr AI▲1,196
Auto-generate B-roll visuals for content marketing videos
AI · 2024 · jupitrr.com
IA
InstantDB – A Modern Firebase▲1,145
Hey there HN! We’re Joe and Stopa, and today we’re open sourcing InstantDB, a client-side database that makes it easy to build real-time and collaborative apps like Notion and Figma. Building modern apps these days involves a lot of schleps. For a basic CRUD app you need to spin up servers, wire up endpoints, integrate auth, add permissions, and then marshal data from the backend to the frontend and back again. If you want to deliver a buttery smooth user experience, you’ll need to add optimistic updates and rollbacks. We do these steps over and over for every feature we build, which can…
Dev tools · 2024 · github.com