Robots.txt search engine

What it does

Search engine for robots.txt contents. Indexes the paths and access rules for which enabled you to find hosts that has specific disallow rules and see the most disallowed crawler agents.

• 
  Robots.txt Generator2024 · generaterobotstxt.com · ▲378

  The right robots.txt file for your project

• 
  AI Search Crawler Check2025 · llmseoindex.com · ▲181

  Check if your robots.txt allows ChatGPT, Google Gemini

• RT
  robots.txt as a service, check web crawl rules through an API2019 · robotstxt.io · ▲17
• 
  Robots.txt Generator2024 · konigle.com · ▲7

  Free and powerful robots.txt generator tool

• 
  Robots.txt TesterFeb 2026 · devhackes.tech · ▲2

  Test Googlebot, Bingbot, or any custom user-agent instantly

• 
  Robots.txt Generator 2023 · techcopes.com · ▲18

  Create a robots.txt file for website instantly

• 
  /dev/null▲494

  a social network for devs

  Dev tools · 4d ago · dev-null.sh

• 
  Run full iOS 16.5 firmware on Android Devices▲470

  Contribute to oosexgang/hackios-xnu-kernel development by creating an account on GitHub.

  Dev tools · 22h ago · github.com

• 
  Whiteboard (YC W26) – an open-source IDE for thoughtful software design▲423

  open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.

  Dev tools · 29d ago · github.com

• 
  Reladraw – A diagram language where you decide where to place things▲411

  Contribute to reladraw/reladraw development by creating an account on GitHub.

  Dev tools · 27d ago · github.com

• 
  OpenBot▲362

  Grok Bot alternative: free, local, open-source, multiplayer

  Dev tools · 24d ago · openbot.run

• 
  Talanoa▲346

  Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…

  Dev tools · 22d ago · talanoa.email

• 
  Terms & Conditions Generator by iubenda▲891

  Attorney-approved terms and conditions generator

  Dev tools · 2019 · iubenda.com

• 
  See a Satellite Tonight. No Telescope Required▲863

  Satellites 🛰️ are visible in the night sky, no telescope required, if you know where to look. Find out here.

  Life & fun · 2019 · james.darpinian.com

• 
  100,000 Faces▲845

  Free resource of 100k diverse faces generated by AI

  AI · 2019 · generated.photos

• 
  Cyberpunk web design made easy, really easy▲775

  Life & fun · 2019 · augmented-ui.com

• 
  Browse Reddit in 3D▲720

  Life & fun · 2019 · 3dforreddit.com

• AS
  A Senior Engineer's CheckList▲662

  Sharing something that I wrote for my use but I think can be useful to others (I am an engineer with the title of Senior Software Engineer) Visit [1] for a full list (table view gives a lot of options. See "Controls"). Visit [2] for a summary. The source of the list is at [3] 1. https://littleblah.com/post/2019-09-01-senior-engineer-checklist/ 2. https://medium.com/@littleblah/my-top-25-items-in-a-senior-engineers-checklist-c8e9f9f6e3c2 3. https://github.com/littleblah/senior-engineer-checklist Feedbacks/Pull requests…

  Dev tools · 2019