Robots.txt Tester
Test Googlebot, Bingbot, or any custom user-agent instantly
Robots.txt Tester launched on February 22, 2026 with 2 votes, #6,500 of 16,627 launches that month.
What it does
Test whether a URL is allowed or blocked by robots.txt rules for Googlebot and other crawlers.
CSS Minifier instantly compresses and minifies your CSS code by removing whitespace and comments. Try our free tool now for faster website loading speed!
Convert, process, and optimize css minifier properties online. Press execution trigger. Image, Text and Developer formats convert locally. Share your experience with this tool or suggest new features you would like tofrom devhackes.tech
Does a similar job
all alternatives →
AI Search Crawler Check2025 · llmseoindex.com · ▲181Check if your robots.txt allows ChatGPT, Google Gemini
Robots.txt Validator & Testing ToolJul 2026 · loganmosby.com · ▲6Validate robots.txt rules before you publish
- RTrobots.txt as a service, check web crawl rules through an API2019 · robotstxt.io · ▲17
Robots.txt Validator and Testing Tool2025 · digitalmansoor.com · ▲12Fetch and analyze any website's robots.txt file in real-time
CrawlerCheck2025 · crawlercheck.com · ▲16Since the last Show HN 7 months ago, I've shipped some major updates: • Operators Directory – browse 160+ crawlers by company (Google, OpenAI, Anthropic, etc.). See all their bots, safety stats, and official docs in one place. • Enhanced crawler profiles with "Blocking Impact" analysis and common use cases • Added 25+ new AI crawlers including latest models and scrapers • Bulk robots.txt generator – generate allow/block rules for all filtered bots in one click • RFC 9309 strict compliance – mimics Googlebot's exact "Longest Match" logic The tool now helps answer "who owns CCBot?" or…
More dev tools this month
the category →
open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.
Dev tools · 28d ago · github.com
Contribute to reladraw/reladraw development by creating an account on GitHub.
Dev tools · 26d ago · github.com
OpenBot▲362Grok Bot alternative: free, local, open-source, multiplayer
Dev tools · 23d ago · openbot.run
Talanoa▲346Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…
Dev tools · 21d ago · talanoa.email
CoreSpeed▲332One MCP for everything your agents need: apps, memory, tools
Dev tools · 29d ago · corespeed.io
Source: Product Hunt launch ↗
Launched alongside, February 2026
the whole month →
Lovon AI Therapy▲801Feel closer and have more fun with your partner every day!
AI · Feb 2026 · lovon.app
Hello my name is Ben Ward for the past 3 years I have been remastering the financial game Wall Street Raider created by Michael Jenkins originally on DOS in 1986. It has been a rough journey but I finally see the light at the end of the tunnel. I just recently redid the website and thought maybe the full story of how this project came to be would interest you all. Thank you for reading.
Work · Feb 2026 · wallstreetraider.com