Chonkie – A Fast, Lightweight Text Chunking Library for RAG
I built Chonkie because I was tired of rewriting chunking code for RAG applications. Existing libraries were either too bloated (80MB+) or too basic, with no middle ground. Core features: - 21MB default install vs 80-171MB alternatives - 33x faster token chunking than popular alternatives - Supports multiple chunking strategies: token, word, sentence, and semantic - Works with all major tokenizers (transformers, tokenizers, tiktoken) - Zero external dependencies for basic functionality Technical optimizations: - Uses tiktoken with multi-threading for faster tokenization - Implements…
Chonkie – A Fast, Lightweight Text Chunking Library for RAG launched on November 10, 2024 with 199 votes, #208 of 4,990 launches that month and more than 95% of that year's launches. The site was not reachable when checked in September 2026.
In plain words
Chonkie is a lightweight text chunking library designed for retrieval-augmented generation (RAG) applications. It offers four chunking strategies—token, word, sentence, and semantic—while maintaining a 21MB footprint, significantly smaller than existing alternatives. The library achieves 33x faster token chunking through multi-threaded tokenization and runs with zero external dependencies for basic functionality, supporting all major tokenizers. It is built for developers who need efficient text processing without the bloat of heavier options.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
I built Chonkie because I was tired of rewriting chunking code for RAG applications. Existing libraries were either too bloated (80MB+) or too basic, with no middle ground. Core features: - 21MB default install vs 80-171MB alternatives - 33x faster token chunking than popular alternatives - Supports multiple chunking…
Does a similar job
all alternatives →
Chonky – a neural approach for text semantic chunking2025 · github.com · ▲169TLDR: I’ve made a transformer model and a wrapper library that segments text into meaningful semantic chunks. The current text splitting approaches rely on heuristics (although one can use neural embedder to group semantically related sentences). I propose a fully neural approach to semantic chunking. I took the base distilbert model and trained it on a bookcorpus to split concatenated text paragraphs into original paragraphs. Basically it’s a token classification task. Model fine-tuning took day and a half on a 2x1080ti. The library could be used as a text splitter module in a RAG system or…
- ACAdvanced Chunking in JavaScript/TypeScript with Chonkie2025 · ▲10
Hi HN, We’re Shreyash and Bhavnick. We built Chonkie, an open-source library for advanced chunking and embedding of text and code. It was previously Python-only, but we just released a TypeScript version: https://github.com/chonkie-inc/chonkie-ts Many AI projects in JS/TS (like those using Vercel's AI SDK or Mastra) rely on basic text splitters. But better chunking = better retrieval = better performance. That’s what Chonkie is built for. Current native chunkers (in TS): - Code Chunker – handles Python, TypeScript, etc. - Recursive Chunker – rule-based, hierarchical…
Chonky – a neural text semantic chunking goes multilingualOct 2025 · huggingface.co · ▲43TLDR: I’m expanding the family of text-splitting Chonky models with new multilingual model. You can learn more about this neural approach in a previous post: https://news.ycombinator.com/item?id=43652968 Since the release of the first distilbert-based model I’ve released two more models based on a ModernBERT. All these models were pre-trained and fine-tuned primary on English texts. But recently mmBERT(https://huggingface.co/blog/mmbert) has been released. This model pre-trained on massive dataset that contains 1833 languages. So I had an idea of…
Ultra-lightweight chunker library with emoji support2025 · github.com · ▲19I kept rewriting the same array.slice(i, i + n) logic in every project. Eventually, I decided to stop and just make a utility. Chonkify is a small (~870B), zero-dependency JS/TS library for chunking any kind of data: - Works with Array, String, Buffer, Set, Map, Array-like, TypedArray - Handles AsyncIterable (great for batching streams) - Unicode-safe (correctly handles grapheme clusters, emoji, etc.) - ESM-first, TypeScript-ready I mostly made it for myself, but figured others might find it useful too. Would love feedback!
- CCChonkie Cloud – No-nonsense chunking now on the the cloud2025 · cloud.chonkie.ai · ▲6
Hey HN, We’re the developers of Chonkie, a powerful and easy-to-use chunking library. Last November, we introduced Chonkie OSS here on HN (https://news.ycombinator.com/item?id=42100819) and got some fantastic feedback. Today, we’re excited to share Chonkie Cloud, a fully hosted service for chunking, along with new tools to make building and debugging RAG systems easier than ever. --- A quick recap for context: We launched Chonkie as an open-source project late last year. A few weeks ago, we decided to go full-time on it. Unfortunately, this shift wasn’t as smooth as we had…
FastGraphRAG – Better RAG using good old PageRank2024 · github.com · ▲457Hey there HN! We’re Antonio, Luca, and Yuhang, and we’re excited to introduce Fast GraphRAG, an open-source RAG approach that leverages knowledge graphs and the 25 years old PageRank for better information retrieval and reasoning. Building a good RAG pipeline these days takes a lot of manual optimizations. Most engineers intuitively start from naive RAG: throw everything in a vector database and hope that semantic search is powerful enough. This can work for use cases where accuracy isn’t too important and hallucinations are tolerable, but it doesn’t work for more difficult queries that…
Questions this answers
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 21d ago · stillwet.art
PixVerse R2▲381A real-time world model you can explore and change
AI · 30d ago · world.pixverse.video
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, November 2024
the whole month →
Lovable▲1,477Hey HN! Previous CERN physicist turned hacker here. We've developed a way to make AI coding actually work by systematically identifying and fixing places where LLMs typically fail in full-stack development. Today we're launching as Lovable (previously gptengineer.app) since it's such a big change. The problem? AI writing code typically make small mistakes and then get stuck. Those who tried know the frustration. We fixed most of this by mapping out where LLMs fail in full-stack dev and engineering around those pitfalls with prompt chains. Thanks to this, in all comparisons I found with: v0,…
AI · 2024 · lovable.dev
Postiz▲1,188Streamline your social media with Postiz. Schedule posts, analyze performance, and manage all accounts in one place.
Dev tools · 2024 · postiz.com
I wasn't quite sure if this qualified as "Show HN" given you can't really download it and try it out. However, dang said[0]: > If it's hardware or something that's not so easy to try out over the internet, find a different way to show how it actually works—a video, for example, or a detailed post with photos. Hopefully I did that? Additionally, I've put code and a detailed guide for the netboot computer management setup on GitHub: https://github.com/kentonv/lanparty Anyway, if this shouldn't have been Show HN, I apologize! [0]…
Dev tools · 2024 · lanparty.house
Raycast Notes▲1,058Unlock a new level productivity with AI, Cloud Sync and more
Work · 2024 · raycast.com
