CrawlerCheck

What it does

Use CrawlerCheck to instantly verify Googlebot, ChatGPT, and other user-agents' access to your site by analyzing robots.txt, meta tags, and HTTP headers.

Free website crawl test for Googlebot, Bingbot, ChatGPT-User, Claude, and Perplexity. Test crawl website access and see if your robots.txt blocks important crawlers.

JavaScript is required for CrawlerCheck to function. Please enable JavaScript in your browser. Please enter a valid URL containing the https:// protocol to check crawler access. Enter a URL and press Check to verify crawler access. Use Reset to clear the form. Enter any URL to instantly check if major search engine and AI crawlers like Googlebot and ChatGPT are allowed or blocked from accessing your website URL and pages. View Supported Crawlers & User Agents Accidentally blocking the wrong bot can hurt your rankings. Verify your crawl rules and optimize your website to ensure improved visibility on search engines and AI bots. How to Improve SEO Visibility Common Issues & Fixes It analyzes…from crawlercheck.com

In the maker’s words, at launch

Since the last Show HN 7 months ago, I've shipped some major updates: • Operators Directory – browse 160+ crawlers by company (Google, OpenAI, Anthropic, etc.). See all their bots, safety stats, and official docs in one place. • Enhanced crawler profiles with "Blocking Impact" analysis and common use cases • Added…

• 
  AI Search Crawler Check2025 · llmseoindex.com · ▲181

  Check if your robots.txt allows ChatGPT, Google Gemini

• 
  Free AI SEO AuditorMay 2026 · freeaiseoaudit.com · ▲166

  Audit your site for the AI search era. 100% Open Source

• 
  Checkbot for Chrome Beta2018 · checkbot.io · ▲229

  Check the SEO, speed & security of 100s of pages in a click!

• 
  Checkbot for Chrome – web crawler that tests for web best practices2018 · checkbot.io · ▲83

  Checkbot finds SEO, speed & security problems before your website visitors do. Tests for broken links, duplicate content, invalid HTML/CSS/JavaScript, insecure pages & more. Test 100s of pages at once including local sites to keep on top of problems & stop critical issues going live.

• 
  AI Search IndexJan 2026 · aisearchindex.com · ▲12

  Your website has new visitors. AI agents from ChatGPT, Perplexity, Claude and other LLMs are reading your content. Track the share of AI agent vs. human traffic and optimize for the agentic web.

• 
  GeoPromptTrackerJul 2026 · geoprompttracker.com · ▲4

  Directory of all 28 AI crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and more — with robots.txt rules to allow or block each one.

• 
  /dev/null▲490

  a social network for devs

  Dev tools · 3d ago · dev-null.sh

• 
  Whiteboard (YC W26) – an open-source IDE for thoughtful software design▲423

  open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.

  Dev tools · 28d ago · github.com

• 
  Reladraw – A diagram language where you decide where to place things▲411

  Contribute to reladraw/reladraw development by creating an account on GitHub.

  Dev tools · 26d ago · github.com

• 
  OpenBot▲362

  Grok Bot alternative: free, local, open-source, multiplayer

  Dev tools · 23d ago · openbot.run

• 
  Talanoa▲346

  Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…

  Dev tools · 21d ago · talanoa.email

• 
  CoreSpeed▲332

  One MCP for everything your agents need: apps, memory, tools

  Dev tools · 29d ago · corespeed.io

• 
  Clueso▲1,300

  Create stunning product videos in minutes with AI

  AI · 2025 · clueso.io

• 
  Trickle - Magic Canvas ▲1,064

  Hey HN! Introducing Trickle Magic Canvas: the world’s first Agentic Canvas, where you can visually co-create apps and websites with AI, from idea to production. Why is a canvas better? Most AI tools today rely on linear chats or text prompts. We believe that limits creativity and clarity. Canvas changes the game. 1. Every version & page at a glance. Each page and sub-page lives on a timeline. You can quickly browse, edit, and preview progress like a living, visual document — no more digging through chat logs or scattered states. 2. The canvas is the context. Unlike linear chats, you can…

  AI · 2025 · trickle.so

• TB
  Tinder but it's only pictures of my wife and I can only swipe right▲1,030

  Life & fun · 2025 · trytender.app

• 
  Ten years of running every day, visualized▲958

  Today marks ten years, 3653 consecutive days, of running at least one mile every day under the USRSA rules [1]. To celebrate, I built an interactive dashboard that turns a decade of GPX files into charts you can explore. Running has truly changed my life: I've made lifelong friends, explored beautiful places, and more importantly invested into my own health and fitness, which I'm starting to see the positive benefits as I get older. The stack is pretty simple: a NextJS app, with a Postgres database to keep all my running data, and all the stats are pre-computed and cached in Redis, so I…

  Dev tools · 2025 · nodaysoff.run

• 
  Draw a fish and watch it swim with the others▲935

  Made this website as an exercise in vibe-coding and GCP. It was posted about a few times around the internet, on sites like Morning Brew (https://www.morningbrew.com/issues/business-buzzkill), MetaFilter (https://www.metafilter.com/209703/Draw-A-Fish), boingboing.net, etc. I think it's cute! I built a basic CNN trained against penises and swastikas, and then anything that doesn't hit the 63% confidence score gets sent to a mod queue, a vibe-coded fish-tinder…

  AI · 2025 · drawafish.com

• 
  Use Their ID – Use your local UK MP’s ID for the Online Safety Act▲862

  Hi HN - I made a site that takes a UK postcode, grabs the local MP's information and generates an AI mockup of what their ID might look like. It's a small, silly protest at the stupidity of the Online Safety Act that just came into force. edit - My open AI credits got hugged to death, please use a known postcode (like one from Kier Starmer's constituency, WC2B6NH) in the meantime.

  AI · 2025 · use-their-id.com