Life & fun · alternatives · 2026

24 alternatives to A modular, durable web-crawler for Clojure

:racehorse:✈️ Pegasus is a scalable, modular, polite web-crawler for Clojure - shriphani/pegasus

Pegasus is a web-crawler library for Clojure developers that emphasizes modularity and durability. It enables users to build scalable crawling systems while respecting server resources through polite behavior. The tool… Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes. A modular, durable web-crawler for Clojure launched in 2016; newer entries below may have overtaken it.

1. 1
   Open sourcing a data flow engine for Clojure/Script▲116

   2019 · domino-clj.github.io · its alternatives →

2. 2
   Chonkie – A Fast, Lightweight Text Chunking Library for RAG▲199

   I built Chonkie because I was tired of rewriting chunking code for RAG applications. Existing libraries were either too bloated (80MB+) or too basic, with no middle ground. Core features: - 21MB default install vs 80-171MB alternatives - 33x faster token chunking than popular alternatives - Supports multiple chunking strategies: token, word, sentence, and semantic - Works with all major tokenizers (transformers, tokenizers, tiktoken) - Zero external dependencies for basic functionality Technical optimizations: - Uses tiktoken with multi-threading for faster tokenization - Implements…

   2024 · github.com · its alternatives →

3. 3DR
   DontBeEvil.rip: Search, for developers (API, expressions, CLI)▲269

   I'd like to invite everyone to try out DontBeEvil.rip, an experimental search engine for developers. tl;dr $ alias rip="curl -G -H 'Accept: text/plain' --url https://dontbeevil.rip/search --data-urlencode " $ rip 'q=Heartbleed bug' DontBeEvil.rip is a year long experiment to see if a small team can build a developer-focused search engine that is self-sustaining on $10 monthly subscriptions. It works by only indexing high-quality resources that are relevant to developers. You won't get useless listicles because we'll never crawl them. Relevant urls are harvested from HN,…

   2022 · its alternatives →

4. 4
   Tantivy▲68

   A full-text, horse-speed search engine library in Rust

   2022 · github.com · its alternatives →

5. 5
   Turn any website into a knowledge base for LLMs▲305

   I built this tool because I wanted a way to just take a bunch of URLs or domains, and query their content in RAG applications. It takes away the pain of crawling, extracting content, chunking, vectorizing, and updating periodically. I'm curious to see if it can be useful to others. I meant to launch this six months ago but life got in the way...

   2024 · embedding.io · its alternatives →

6. 6
   Sukhoi – A flexible and extensible Webcrawler in Python▲131

   Minimalist and powerful Web Crawler. Contribute to untwisted/sukhoi development by creating an account on GitHub.

   2017 · github.com · its alternatives →

7. 7
   Kinase▲159

   A framework for building web scrapers for Chrome

   2017 · b12.io · its alternatives →

8. 8HN
   Hacker News Clone – Clojure, GraphQL Learning Project▲179

   I have ported "The Fullstack Tutorial for GraphQL" series, https://www.howtographql.com/, from Javascript to Clojure/script. Blog post: https://promesante.github.io/2019/08/14/clojure_graphql_full... Source code GitHub repo: https://github.com/promesante/hn-clj-pedestal-re-frame Suggestions and corrections, more than welcome.

   2019 · its alternatives →

9. 9
   Jevcrawl – A web crawler for AI agents with no LLM in the crawl loop▲21

   MCP server and CLI that crawls websites for AI agents: finds the answer page 2x more often than Crawl4AI, 10x cheaper per answer than letting Claude crawl. - RayenMlayeh/jevcrawl

   8d ago · github.com · its alternatives →

10. 10
    Grasp – grep Clojure code using clojure.spec regexes▲123

    Grep Clojure code using clojure.spec regexes. Contribute to borkdude/grasp development by creating an account on GitHub.

    2020 · github.com · its alternatives →

11. 11
    Apify SDK – A scalable web crawling and scraping library for JavaScript▲78

    Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headf

    2018 · github.com · its alternatives →

12. 12
    Open-sourcing our large Clojure(Script) Re-frame/Hasura web app▲21

    SaaS buying and decision platform. Contribute to Vetd-Inc/vetd-app development by creating an account on GitHub.

    2020 · github.com · its alternatives →

13. 13
    Luna is a Clojure domain specific language that translates to regex▲56

    A DSL that translates to regex. Contribute to AbhinavOmprakash/luna development by creating an account on GitHub.

    2021 · github.com · its alternatives →

14. 14
    Runiq, a little Lisp-inspired language that runs on JavaScript▲24

    :aquarius: A Lisp-esque, JS-interpreted scripting language - tsortwehttam/runiq

    2015 · github.com · its alternatives →

15. 15CO
    Coast on Clojure: A full stack framework with a catchy name▲8

    2017 · medium.com · its alternatives →

16. 16RO
    Ridiculously over-engineered my blog▲15

    2016 · ponyfoo.com · its alternatives →

17. 17
    A distributed and persistent web crawler written in Go▲5

    Cross-platform persistent and distributed web crawler :link: - schollz/linkcrawler

    2017 · github.com · its alternatives →

18. 18MC
    Make CRUD apps in real-time with ClojureScript▲16

    A PII-safe service observability and management surface. Classpath-connected, no REST APIs.

    2018 · hyperfiddle.net · its alternatives →

19. 19
    Airgapped Offline RAG – Run LLMs Locally with Llama, Mistral, & Gemini▲9

    I've built an airgapped Retrieval-Augmented Generation (RAG) system for question-answering on documents, running entirely offline with local inference. Using Llama 3, Mistral, and Gemini, this setup allows secure, private NLP on your own machine. Perfect for researchers, data scientists, and developers who need to process sensitive data without cloud dependencies. Built with Llama C++, LangChain, and Streamlit, it supports quantized models and provides a sleek UI for document processing. Check it out, contribute, or suggest new features!

    2024 · github.com · its alternatives →

20. 20
    LLMic▲10

    Force Google & Bing to index your content instantly.

    Dec 2025 · llmic.com · its alternatives →

21. 21
    An example web app (board game) for learning ClojureScript▲8

    A nontrivial example for ClojureScript + Firebase. - jacobobryant/mystery-cows

    2020 · github.com · its alternatives →

22. 22
    Choo – 5kb framework for creating sturdy front end applications▲16

    :steam_locomotive::train: - sturdy 4kb frontend framework - choojs/choo

    2016 · github.com · its alternatives →

23. 23SF
    "Substack for RSS feeds”, built using Clojure and Datomic Ions▲5

    2021 · ab.xyz · its alternatives →

24. 24IB
    I built an SDK that scrambles HTML so scrapers get garbage▲16

    Hey HN -- I'm a solo dev. Built this because I got tired of AI crawlers reading my HTML in plain text while robots.txt did nothing. The core trick: shuffle characters and words in your HTML using a seed, then use CSS (flexbox order, direction: rtl, unicode-bidi) to put them back visually. Browser renders perfectly. textContent returns garbage. On top of that: email/phone RTL obfuscation with decoy characters, AI honeypots that inject prompt instructions into LLM scrapers, clipboard interception, canvas-based image rendering (no img src in DOM), robots.txt blocking 30+ AI crawlers, and…

    Mar 2026 · obscrd.dev · its alternatives →

Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →