A modular, durable web-crawler for Clojure
:racehorse:✈️ Pegasus is a scalable, modular, polite web-crawler for Clojure - shriphani/pegasus
A modular, durable web-crawler for Clojure launched on January 25, 2016 with 29 votes, #1,962 of 4,522 launches that month and more than 75% of that year's launches. The site was still online in September 2026, 10 years on.
In plain words
Pegasus is a web-crawler library for Clojure developers that emphasizes modularity and durability. It enables users to build scalable crawling systems while respecting server resources through polite behavior. The tool is designed for developers who need to systematically extract data from websites within a functional programming framework.
written from the facts on this page · September 2026
Does a similar job
all alternatives →
Open sourcing a data flow engine for Clojure/Script2019 · domino-clj.github.io · ▲116
Chonkie – A Fast, Lightweight Text Chunking Library for RAG2024 · github.com · ▲199I built Chonkie because I was tired of rewriting chunking code for RAG applications. Existing libraries were either too bloated (80MB+) or too basic, with no middle ground. Core features: - 21MB default install vs 80-171MB alternatives - 33x faster token chunking than popular alternatives - Supports multiple chunking strategies: token, word, sentence, and semantic - Works with all major tokenizers (transformers, tokenizers, tiktoken) - Zero external dependencies for basic functionality Technical optimizations: - Uses tiktoken with multi-threading for faster tokenization - Implements…
- DRDontBeEvil.rip: Search, for developers (API, expressions, CLI)2022 · ▲269
I'd like to invite everyone to try out DontBeEvil.rip, an experimental search engine for developers. tl;dr $ alias rip="curl -G -H 'Accept: text/plain' --url https://dontbeevil.rip/search --data-urlencode " $ rip 'q=Heartbleed bug' DontBeEvil.rip is a year long experiment to see if a small team can build a developer-focused search engine that is self-sustaining on $10 monthly subscriptions. It works by only indexing high-quality resources that are relevant to developers. You won't get useless listicles because we'll never crawl them. Relevant urls are harvested from HN,…
Sukhoi – A flexible and extensible Webcrawler in Python2017 · github.com · ▲131Minimalist and powerful Web Crawler. Contribute to untwisted/sukhoi development by creating an account on GitHub.
Jevcrawl – A web crawler for AI agents with no LLM in the crawl loop8d ago · github.com · ▲21MCP server and CLI that crawls websites for AI agents: finds the answer page 2x more often than Crawl4AI, 10x cheaper per answer than letting Claude crawl. - RayenMlayeh/jevcrawl
More life & fun this month
the category →
A free city-building game in your browser. You are the mayor. You know what the city needs. Now get it past the zoning board, the neighbors, the appeals court and the next election.
Life & fun · 2d ago · permit.petercai.com
- IV
Life & fun · 14d ago · tanzaniaeducationproject.org
Turn a phone, tablet or TV into a sign. Type a message and it fills the screen: welcome signs, timers, Wi-Fi passwords and more. No app, no account, free.
Life & fun · 15d ago · bigwords.page
Play a free virtual guitar online with your keyboard or by touch. Slow songs down and see every chord, string and strum. Acoustic, classic, electric and bass.
Life & fun · 9d ago · guitoro.com
Carrier settings from iPhone, Pixel and Galaxy firmware, decoded and compared: APNs, VoLTE, 5G and Wi-Fi Calling per carrier, and what each build changed.
Life & fun · 13d ago · carrierexplode.com
Life & fun · 23d ago · space.bl2.net
Source: Show HN post ↗
Launched alongside, January 2016
the whole month →
Website Grader▲768Grade your website's strengths and weaknesses in seconds.
Growth · 2016 · website.grader.com
Clearbit Connect▲723Rapportive on steroids. See data about contacts in Gmail.
Growth · 2016 · clearbit.com
Create a viral waiting list in 60 seconds
Growth · 2016 · app.referralhero.com