Alternatives

Products that do what Web Crawler Directory & Bot Database does

Identify, check, manage unwanted web crawlers and bots.

1. 1
   AI Search Crawler Check▲181

   Check if your robots.txt allows ChatGPT, Google Gemini

   Sep 2025

2. 2
   LLM SEO Index Crawler Check▲157

   Check if your website can get crawled by ChatGPT

   2025

3. 3
   AI Search Index▲12

   Track which AI bots crawl your website

   Jan 2026 · aisearchindex.com

4. 4
   Known Agents▲95

   Track the bots and AI agents crawling your website

   May 2026 · knownagents.com

5. 5
   There Is A Bot For That▲219

   Search engine for bots

   2017

6. 6
   Crawlee▲229

   Build reliable web scrapers and robots, fast!

   2022

7. 7
   Robots.txt Generator▲389

   The right robots.txt file for your project

   2024

8. 8
   Bot Hunt▲317

   Largest directory of 'Chatbots' on the internet

   2024

9. 9CF
   Checkbot for Chrome – web crawler that tests for web best practices▲83

   2018 · checkbot.io

10. 10
    Checkbot for Chrome Beta▲229

    Check the SEO, speed & security of 100s of pages in a click!

    2018

11. 11
    BotPages▲94

    Find, discover, and promote bots.

    2016

12. 12
    Agent Directory | Rankly▲1

    Largest Catalog of 4000+ Agents, Bots and Crawlers in web

    May 2026 · tryrankly.com

13. 13
    IndexDecode▲6

    Check if AI crawlers can find, read, and cite your site

    Jul 2026 · indexdecode.com

14. 14
    Gobot▲184

    The Citymapper bot

    2016

15. 15
    ScrapingBot▲170

    The API you need for efficient scraping!

    2019

16. 16
    Bot Check▲7

    Protect your blog and creative work from AI scrapers

    2025

17. 17
    BotSpeak database▲120

    1 human, 200+ bots, 892 bot dialogs

    2016

18. 18
    GPT Crawler▲98

    Crawl a site to generate knowledge files to create a GPT

    2023

19. 19
    Crawl AI▲116

    Build Your Own AI With One Prompt

    2025

20. 20
    PHPBot▲136

    The ultimate PHP Snippet bot

    2017

21. 21CO
    Crawlab: Open-Source Web Crawler Admin Platform That Runs Any Language▲116

    2019 · github.com

22. 22
    .BOT▲80

    The online space for bots

    2017

23. 23
    SEO Scrubbox▲13

    Technical SEO checks for the AI search era

    Jun 2026 · chromewebstore.google.com

24. 24
    GeoPromptTracker▲4

    19 free tools to make your site visible to AI search

    Jul 2026 · geoprompttracker.com

Ranked by how close each launch is in meaning, then by votes. Refine with a description →