CrawlFence

What it does

CrawlFence offers advanced bot protection for small and medium-sized enterprises, safeguarding your digital assets from unauthorized crawlers and malicious bots.

Scan your site for AI-crawler exposure and get clear, SEO-safe steps to protect your content and IP from unwanted AI training.

Scan your site for AI crawler exposure, robots.txt rules, legal notices, hosting setup, and practical protection steps — while preserving normal search visibility. Free first scan. No account required. We only inspect publicly available website information. Re-runs within 72 hours show your previous report. A superficial but real check of your site's public signals. No login, no payment, no access to private data. Detects hosting platform and CDN signals such as Wix, Squarespace, Vercel, Cloudflare, and more. Looks for a robots.txt file at your site root and inspects its directives. Checks for rules covering GPTBot, Google-Extended, CCBot, ClaudeBot, PerplexityBot, and others. Confirms that…from crawlfence.com

• 
  Crawlee2022 · crawlee.dev · ▲229

  Build reliable web scrapers and robots, fast!

• 
  Crawly2018 · ▲234

  Uptime, performance, SSL certificate and asset monitoring.

• 
  Crawl AI2025 · crawlai.org · ▲116

  Build Your Own AI With One Prompt

• 
  Known AgentsMay 2026 · knownagents.com · ▲95

  Track the bots and AI agents crawling your website

• 
  ClicKarma.io2023 · clickarma.io · ▲90

  Don’t let bots click on your ads

• 
  CrawlrLabs2024 · ▲68

  AI-optimized competitive pricing solutions for E-commerce

• 
  /dev/null▲487

  a social network for devs

  Dev tools · 3d ago · dev-null.sh

• 
  Whiteboard (YC W26) – an open-source IDE for thoughtful software design▲423

  open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.

  Dev tools · 28d ago · github.com

• 
  Reladraw – A diagram language where you decide where to place things▲411

  Contribute to reladraw/reladraw development by creating an account on GitHub.

  Dev tools · 26d ago · github.com

• 
  OpenBot▲362

  Grok Bot alternative: free, local, open-source, multiplayer

  Dev tools · 23d ago · openbot.run

• 
  Talanoa▲346

  Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…

  Dev tools · 21d ago · talanoa.email

• 
  CoreSpeed▲332

  One MCP for everything your agents need: apps, memory, tools

  Dev tools · 28d ago · corespeed.io

• 
  buzzabout▲1,267

  Audience insights from 1B+ online discussions in 2 mins

  AI · 2024 · buzzabout.ai

• 
  bolt.new▲1,230

  Build and scale high-performing websites & apps using your words. Join millions and start building today.

  AI · 2024 · bolt.new

• 
  Feta▲1,217

  Run smarter stand-ups, build better products

  AI · 2024 · feta.io

• 
  Trag▲955

  AI code review companion

  AI · 2024 · usetrag.com

• 
  Softr for Notion▲923

  Turn Notion databases into portals & apps with no code

  Dev tools · 2024 · softr.io

• 
  General Collaboration▲910

  One inbox for all your work discussions

  Work · 2024 · generalcollaboration.com