Robots.txt Tester

What it does

Robots.txt Tester by DevHackes lets you check if your robots.txt file is blocking the pages you actually want crawled — or worse, letting bots crawl what you meant to hide. Paste your file or fetch it live from any domain, pick a user-agent (Googlebot, Bingbot, AI crawlers), enter your URLs, and instantly see Allowed vs Blocked with the exact rule responsible. No signup, no limits — built for developers and SEOs who want quick, accurate answers.

Test your robots.txt crawl directives against standard user agents to verify crawl paths permissions.

• 
  AI Search Crawler CheckSep 2025 · ▲181

  Check if your robots.txt allows ChatGPT, Google Gemini

• 
  LLM SEO Index Crawler Check2025 · ▲157

  Check if your website can get crawled by ChatGPT

• 
  Checkbot for Chrome Beta2018 · ▲229

  Check the SEO, speed & security of 100s of pages in a click!

• RT
  robots.txt as a service, check web crawl rules through an API2019 · robotstxt.io · ▲17
• 
  Robotic Tape2019 · ▲232

  Quickly build an API for any website, visually.

• RT
  Robots.txt generator for blocking AI crawlers2024 · aisearchwatch.com · ▲9

• 
  Dograh▲592

  The open source VAPI alternative

  Dev tools · 25d ago · dograh.com

• 
  Meridian▲530

  Don't let your work go unnoticed. Get promoted!

  Dev tools · 20d ago · meridiona.com

• 
  x1▲516

  Lovable for iPhone apps go from idea to App Store

  Dev tools · 11d ago · x1.new

• 
  The GTM Co-Founder▲402

  Open-source GTM skills for technical founders

  Dev tools · 29d ago · gtmcofounder.com

• 
  Open-Source eInk Bike Computer▲384

  OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.

  Dev tools · 1d ago · opentrailpaper.com

• 
  Nuphos▲380

  The AI-Native DevOps Workspace.

  Dev tools · 24d ago · nuphos.ai

• 
  Open-Source eInk Bike Computer▲384

  OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.

  Dev tools · 1d ago · opentrailpaper.com

• 
  Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s▲237

  Run Qwen3.8-Flash-Next (125B MoE, 104 GB at 4-bit) on Macs with a fraction of that RAM by streaming experts from SSD. MLX + Swift, Ollama-compatible API. - carloslfu/slotstream

  Dev tools · 4d ago · github.com

• 
  TERMy – A fast terminal assistant that does not use LLMs▲185

  NPC-Forge is a framework for building conversational agents that run on the CPU without relying on machine learning or LLMs. - NPC-Forge/docs/development.md at main · gioblu/NPC-Forge

  AI · 2d ago · github.com

• 
  Weedout – Safari extension that hides YouTube AI-labeled videos▲182

  Weedout is a Safari extension that filters out videos YouTube labels as “Made with AI”.

  AI · 4d ago · masteranza.github.io

• 
  HN Match Maker – Matching "Who Wants to Be Hired?" With "Who's Hiring?"▲123

  Find your next hire or your next role from Hacker News monthly threads. AI-powered matching between candidates and job postings.

  AI · 4d ago · hnmatchmaker.com

• 
  OwnTime – a chess clock for your day's priorities▲110

  I made OwnTime to balance between competing priorities in my life. The idea is based on two main influences: the concept of "roles" from "The 5 Choices" (2015), and my repeated failure to effectively implement time blocking due to the necessary flexibility in my role. The app allows you to define time budgets, which are not much more than a few mutually exclusive countdown timers. The UX is essentially that of a chess clock for an arbitrary number of players with configurable time. The whole point of the app is to kick you out of the running role/priority when its time is up. One…

  Life & fun · 4d ago · owntime.app