Chonky – a neural approach for text semantic chunking
TLDR: I’ve made a transformer model and a wrapper library that segments text into meaningful semantic chunks. The current text splitting approaches rely on heuristics (although one can use neural embedder to group semantically related sentences). I propose a fully neural approach to semantic chunking. I took the base distilbert model and trained it on a bookcorpus to split concatenated text paragraphs into original paragraphs. Basically it’s a token classification task. Model fine-tuning took day and a half on a 2x1080ti. The library could be used as a text splitter module in a RAG system or…
Chonky – a neural approach for text semantic chunking launched on April 11, 2025 with 169 votes, #236 of 7,818 launches that month and more than 95% of that year's launches. The site was still online in September 2026, 1 year on.
In plain words
Chonky is a neural text chunking tool that segments documents into meaningful semantic chunks using a transformer model trained on the BookCorpus dataset. Unlike traditional heuristic-based approaches, it uses a fully neural token classification method to identify natural paragraph boundaries in raw text. The library is designed for use in retrieval-augmented generation systems, transcript processing, and other applications requiring intelligent text segmentation. It leverages a fine-tuned DistilBERT model to improve upon conventional splitting methods.
written from the facts on this page · September 2026
From the sources
Fully neural approach for text chunking. Contribute to mirth/chonky development by creating an account on GitHub.
Chonky is a Python library that intelligently segments text into meaningful semantic chunks using a fine-tuned transformer model. This library can be used in the RAG systems. pip install chonky Usage: from chonky import ParagraphSplitter # on the first run it will download the transformer model splitter = ParagraphSplitter ( device = "cpu" ) # Or you can select the model # splitter = ParagraphSplitter( # model_id="mirth/chonky_modernbert_base_1", # device="cpu" # ) text = ( "Before college the two main things I worked on, outside of school, were writing and programming. " "I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short…from github.com
In the maker’s words, at launch
TLDR: I’ve made a transformer model and a wrapper library that segments text into meaningful semantic chunks. The current text splitting approaches rely on heuristics (although one can use neural embedder to group semantically related sentences). I propose a fully neural approach to semantic chunking. I took the base…
Does a similar job
all alternatives →
Chonky – a neural text semantic chunking goes multilingualOct 2025 · huggingface.co · ▲43TLDR: I’m expanding the family of text-splitting Chonky models with new multilingual model. You can learn more about this neural approach in a previous post: https://news.ycombinator.com/item?id=43652968 Since the release of the first distilbert-based model I’ve released two more models based on a ModernBERT. All these models were pre-trained and fine-tuned primary on English texts. But recently mmBERT(https://huggingface.co/blog/mmbert) has been released. This model pre-trained on massive dataset that contains 1833 languages. So I had an idea of…
Chonkie – A Fast, Lightweight Text Chunking Library for RAG2024 · github.com · ▲199I built Chonkie because I was tired of rewriting chunking code for RAG applications. Existing libraries were either too bloated (80MB+) or too basic, with no middle ground. Core features: - 21MB default install vs 80-171MB alternatives - 33x faster token chunking than popular alternatives - Supports multiple chunking strategies: token, word, sentence, and semantic - Works with all major tokenizers (transformers, tokenizers, tiktoken) - Zero external dependencies for basic functionality Technical optimizations: - Uses tiktoken with multi-threading for faster tokenization - Implements…
I modeled the Voynich Manuscript with SBERT to test for structure2025 · github.com · ▲381I built this project as a way to learn more about NLP by applying it to something weird and unsolved. The Voynich Manuscript is a 15th-century book written in an unknown script. No one’s been able to translate it, and many think it’s a hoax, a cipher, or a constructed language. I wasn’t trying to decode it — I just wanted to see: does it behave like a structured language? I stripped a handful of common suffix-like endings (aiin, dy, etc.) to isolate what looked like root forms. I know that’s a strong assumption — I call it out directly in the repo — but it helped clarify the clustering. From…
- ACAdvanced Chunking in JavaScript/TypeScript with Chonkie2025 · ▲10
Hi HN, We’re Shreyash and Bhavnick. We built Chonkie, an open-source library for advanced chunking and embedding of text and code. It was previously Python-only, but we just released a TypeScript version: https://github.com/chonkie-inc/chonkie-ts Many AI projects in JS/TS (like those using Vercel's AI SDK or Mastra) rely on basic text splitters. But better chunking = better retrieval = better performance. That’s what Chonkie is built for. Current native chunkers (in TS): - Code Chunker – handles Python, TypeScript, etc. - Recursive Chunker – rule-based, hierarchical…
Questions this answers
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 21d ago · stillwet.art
PixVerse R2▲381A real-time world model you can explore and change
AI · 30d ago · world.pixverse.video
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, April 2025
the whole month →
Hi everyone, I built PyXL — a hardware processor that executes a custom assembly generated from Python programs, without using a traditional interpreter or virtual machine. It compiles Python -> CPython Bytecode -> Instruction set designed for direct hardware execution. I’m sharing an early benchmark: a GPIO test where PyXL achieves a 480ns round-trip toggle — compared to 14-25 micro seconds on a MicroPython Pyboard - even though PyXL runs at a lower clock (100MHz vs. 168MHz). The design is stack-based, fully pipelined, and preserves Python's dynamic typing without static type restrictions.…
Dev tools · 2025 · runpyxl.com
The Unsure Calculator is an online tool that lets you calculate with numbers you’re not sure about.
Life & fun · 2025 · filiph.github.io
https://the-pocket.github.io/Tutorial-Codebase-Knowledge/
AI · 2025 · github.com
