Alternatives
Products that do what Chonkie – A Fast, Lightweight Text Chunking Library for RAG does
I built Chonkie because I was tired of rewriting chunking code for RAG applications. Existing libraries were either too bloated (80MB+) or too basic, with no middle ground. Core features: - 21MB default install vs 80-171MB alternatives - 33x faster token chunking than popular alternatives - Supports multiple chunking strategies: token, word, sentence, and semantic - Works with all major tokenizers (transformers, tokenizers, tiktoken) - Zero external dependencies for basic functionality Technical optimizations: - Uses tiktoken with multi-threading for faster tokenization - Implements…
- 1CA
TLDR: I’ve made a transformer model and a wrapper library that segments text into meaningful semantic chunks. The current text splitting approaches rely on heuristics (although one can use neural embedder to group semantically related sentences). I propose a fully neural approach to semantic chunking. I took the base distilbert model and trained it on a bookcorpus to split concatenated text paragraphs into original paragraphs. Basically it’s a token classification task. Model fine-tuning took day and a half on a 2x1080ti. The library could be used as a text splitter module in a RAG system or…
2025 · github.com
- 2AC
Hi HN, We’re Shreyash and Bhavnick. We built Chonkie, an open-source library for advanced chunking and embedding of text and code. It was previously Python-only, but we just released a TypeScript version: https://github.com/chonkie-inc/chonkie-ts Many AI projects in JS/TS (like those using Vercel's AI SDK or Mastra) rely on basic text splitters. But better chunking = better retrieval = better performance. That’s what Chonkie is built for. Current native chunkers (in TS): - Code Chunker – handles Python, TypeScript, etc. - Recursive Chunker – rule-based, hierarchical…
2025
- 3CA
TLDR: I’m expanding the family of text-splitting Chonky models with new multilingual model. You can learn more about this neural approach in a previous post: https://news.ycombinator.com/item?id=43652968 Since the release of the first distilbert-based model I’ve released two more models based on a ModernBERT. All these models were pre-trained and fine-tuned primary on English texts. But recently mmBERT(https://huggingface.co/blog/mmbert) has been released. This model pre-trained on massive dataset that contains 1833 languages. So I had an idea of…
Oct 2025 · huggingface.co
- 4UL
I kept rewriting the same array.slice(i, i + n) logic in every project. Eventually, I decided to stop and just make a utility. Chonkify is a small (~870B), zero-dependency JS/TS library for chunking any kind of data: - Works with Array, String, Buffer, Set, Map, Array-like, TypedArray - Handles AsyncIterable (great for batching streams) - Unicode-safe (correctly handles grapheme clusters, emoji, etc.) - ESM-first, TypeScript-ready I mostly made it for myself, but figured others might find it useful too. Would love feedback!
2025 · github.com
- 5CC
Hey HN, We’re the developers of Chonkie, a powerful and easy-to-use chunking library. Last November, we introduced Chonkie OSS here on HN (https://news.ycombinator.com/item?id=42100819) and got some fantastic feedback. Today, we’re excited to share Chonkie Cloud, a fully hosted service for chunking, along with new tools to make building and debugging RAG systems easier than ever. --- A quick recap for context: We launched Chonkie as an open-source project late last year. A few weeks ago, we decided to go full-time on it. Unfortunately, this shift wasn’t as smooth as we had…
2025 · cloud.chonkie.ai
- 6FB
Hey there HN! We’re Antonio, Luca, and Yuhang, and we’re excited to introduce Fast GraphRAG, an open-source RAG approach that leverages knowledge graphs and the 25 years old PageRank for better information retrieval and reasoning. Building a good RAG pipeline these days takes a lot of manual optimizations. Most engineers intuitively start from naive RAG: throw everything in a vector database and hope that semantic search is powerful enough. This can work for use cases where accuracy isn’t too important and hallucinations are tolerable, but it doesn’t work for more difficult queries that…
2024 · github.com
- 7GA
Hi all, I've talked about glidesort a few times on HN already, but it's finally ready for release. If you have any questions, feel free to ask. An academic paper on glidesort that goes into a lot more detail than the readme is upcoming, but is not ready yet. I will be giving a talk on glidesort tomorrow at FOSDEM 2023 in the Rust Devroom at 16:10, you can seek me out there as well. In other news, I am leaving academia soon, so if you have interesting (Rust) jobs the coming months feel free to approach me.
2023 · github.com
- 8AH
Long-simmering side project that is finally ready to see the light. HAMTs are a cool persistent data structure and implementing one has been a lot of fun. Beyond the code, there is likely some value in the extensive and largely complete implementation docs; basic benchmarks are linked in the README, too. Kind of aiming to be "the libavl for HAMTs". That is obviously a high and aspirational bar but a distinct possibility if it stirs up a little interest and/or contribution. Anyways, it's time for this to go out, collect feedback and maybe even some use outside of toy projects. Let me…
2023 · github.com
- 9IM
I built this project as a way to learn more about NLP by applying it to something weird and unsolved. The Voynich Manuscript is a 15th-century book written in an unknown script. No one’s been able to translate it, and many think it’s a hoax, a cipher, or a constructed language. I wasn’t trying to decode it — I just wanted to see: does it behave like a structured language? I stripped a handful of common suffix-like endings (aiin, dy, etc.) to isolate what looked like root forms. I know that’s a strong assumption — I call it out directly in the repo — but it helped clarify the clustering. From…
2025 · github.com
- 10TA
I built this tool because I wanted a way to just take a bunch of URLs or domains, and query their content in RAG applications. It takes away the pain of crawling, extracting content, chunking, vectorizing, and updating periodically. I'm curious to see if it can be useful to others. I meant to launch this six months ago but life got in the way...
2024 · embedding.io
- 11DA
I've built an advanced RAG (Retrieval-Augmented Generation) pipeline from scratch to demystify the complex mechanics of modern LLM-powered Question Answering systems. This repository features: -- An implementation of a sub-question query engine from scratch to answer complex user questions. -- Illustrative explanations that unveil the inner workings of the system. -- An analysis of the challenges I faced while working with the system, like prompt engineering and cost estimation. -- Qualitative comparison with similar frameworks like LlamaIndex, offering a broader perspective. Key Takeaway:…
2023 · github.com
- 12DR
I'd like to invite everyone to try out DontBeEvil.rip, an experimental search engine for developers. tl;dr $ alias rip="curl -G -H 'Accept: text/plain' --url https://dontbeevil.rip/search --data-urlencode " $ rip 'q=Heartbleed bug' DontBeEvil.rip is a year long experiment to see if a small team can build a developer-focused search engine that is self-sustaining on $10 monthly subscriptions. It works by only indexing high-quality resources that are relevant to developers. You won't get useless listicles because we'll never crawl them. Relevant urls are harvested from HN,…
2022
- 13BS
Introducing Biblos, a simple tool for semantic search and summarization of Bible passages. Leveraging Chroma for vector search with BAAI BGE embeddings, semantically find related verses across the Bible. The tool employs Anthropic's Claude LLM model for generating high-quality summaries of retrieved passages, contextualizing your search topic. Built on a Retrieval Augmented Generation (RAG) architecture, the app implements a simple Streamlit Web UI using Python. Deployed using render.com, the app is available at https://biblos.app Note: Search by just topic/keywords, e.g.…
2023 · github.com
- 14RA
RAGLite is a Python package for building Retrieval-Augmented Generation (RAG) applications. RAG applications can be magical when they work well, but anyone who has built one knows how much the output quality depends on the quality of retrieval and augmentation. With RAGLite, we set out to unhobble RAG by mapping out all of its subproblems and implementing the best solutions to those subproblems. For example, RAGLite solves the chunking problem by partitioning documents in provably optimal level 4 semantic chunks. Another unique contribution is its optimal closed-form linear query adapter…
2024 · github.com
- 15CA
ChunkHound’s goal is simple: local-first codebase intelligence that helps you pull deep, core-dev-level insights on demand, generate always-up-to-date docs, and scale from small repos to enterprise monorepos — while staying free + open source and provider-agnostic (VoyageAI / OpenAI / Qwen3, Anthropic / OpenAI / Gemini / Grok, and more). I’d love your feedback — and if you have, thank you for being part of the journey!
Jan 2026 · github.com
- 16
- 17
- 18SA
Hi HN, I’m Tullie, founder of Shaped. Previously, I was a researcher at Meta AI, worked on ranking for Instagram Reels, and was a contributor to PyTorch Lightning. We built ShapedQL because we noticed that while retrieval (finding 1,000 items) has been commoditized by vector DBs, ranking (finding the best 10 items) is still an infrastructure problem. To build a decent for you feed or a RAG system with long-term memory, you usually have to put together a vector DB (Pinecone/Milvus), a feature store (Redis), an inference service, and thousands of lines of Python to handle business logic…
Jan 2026 · playground.shaped.ai
- 19CT
I had been looking to try <500M parameter language models but you wouldn't find an API to try them anywhere, so I built this cloudflare hosted static website that hosts weights and built an inference runtime for these models that uses WebGPU and runs inference from your browser. These are only so useful in a multi-turn conversation but it's still interesting to see what you can pack in a <250mb model. I tried using ONNX versions earlier, but there were too many quirks of using them with language models and the TPS wasn't too impressive. Inspired by svenflow/webgpu-gemma, I put my codex…
May 2026 · chonklm.com
- 20GA
Hi HN, I have been working with regulation-heavy documents lately, and one thing kept bothering me. Flat RAG pipelines often fail to retrieve related articles together, even when they are clearly connected through references, definitions, or clauses. After trying several RAG setups, I subjectively felt that GraphRAG was a better mental model for this kind of data. The Microsoft GraphRAG paper and reference implementation were helpful starting points. However, in practice, I found one recurring friction point: graph storage and vector indexing are usually handled by separate systems, which…
Jan 2026 · github.com
- 21RC
Nov 2025 · github.com
- 22DE
Hey! I wanted to share a tool I've been working on. It's still very early and a work in progress, but I've found it incredibly helpful when working with Claude and OpenAI's models. What it does: I created a Python script that dumps your entire Git repository into a single file. This makes it much easier to use with Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) systems. Key Features: - Respects .gitignore patterns - Generates a tree-like directory structure - Includes file contents for all non-excluded files - Customizable file type filtering Why I find it useful for…
2024
- 23AS
2024 · github.com
- 24RI
Got tired of wiring up vector stores, embedding models, and chunking logic every time I needed RAG. So I built piragi. from piragi import Ragi kb = Ragi(\["./docs", "./code/\*\*/\*.py", "https://api.example.com/docs"\]) answer = kb.ask("How do I deploy this?") That's the entire setup. No API keys required - runs on Ollama + sentence-transformers locally. What it does: - All formats - PDF, Word, Excel, Markdown, code, URLs, images, audio - Auto-updates - watches sources, refreshes in background, zero query latency - Citations - every answer includes sources…
Dec 2025 · pypi.org
Ranked by how close each launch is in meaning, then by votes. Refine with a description →