Chonky – a neural text semantic chunking goes multilingual
TLDR: I’m expanding the family of text-splitting Chonky models with new multilingual model. You can learn more about this neural approach in a previous post: https://news.ycombinator.com/item?id=43652968 Since the release of the first distilbert-based model I’ve released two more models based on a ModernBERT. All these models were pre-trained and fine-tuned primary on English texts. But recently mmBERT(https://huggingface.co/blog/mmbert) has been released. This model pre-trained on massive dataset that contains 1833 languages. So I had an idea of…
In plain words
Chonky is a neural text semantic chunking tool that splits documents into meaningful chunks for processing. It now includes multilingual models trained on 1,833 languages using mmBERT, expanding beyond its original English-focused versions. The models use neural approaches based on DistilBERT and ModernBERT architectures, trained on diverse datasets including books in multiple languages. It is designed for developers and teams working with text processing and natural language applications across different languages.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
TLDR: I’m expanding the family of text-splitting Chonky models with new multilingual model. You can learn more about this neural approach in a previous post: https://news.ycombinator.com/item?id=43652968 Since the release of the first distilbert-based model I’ve released two more models based on a ModernBERT. All these models were pre-trained and fine-tuned primary on English texts. But recently mmBERT(https://huggingface.co/blog/mmbert) has been released. This model pre-trained on massive dataset that contains 1833 languages. So I had an idea of fine-tuning a new multilingual Chonky model. I’ve expanded training dataset (that previously contained bookcorpus and minipile datasets) with Project Gutenberg dataset which contains books in some widespread languages. To make the model more robust for real-world data I’ve removed punctuation for last word for every training chunk with probability of 0.15 (no ablation was made for this technique though). The hard part is evaluation. The real-world data are typically OCR'ed markdown, transcripts of calls, meeting notes etc. and not a clean book paragraphs. I didn’t find such labeled datasets. So I used what I had: already mentioned bookcorpus and Project Gutenberg validation, Paul Graham essays, concatenated 20_newsgroups. I also tried to fine-tune the bigger mmBERT model (mmbert-base) but unfortunately it didn’t go well — metrics are weirdly lower in comparison with a small model. Please give it a try. I'll appreciate a feedback. The new multilingual model: https://huggingface.co/mirth/chonky_mmbert_small_multilingua... All the Chonky models: https://huggingface.co/mirth Chonky wrapper library: https://github.com/mirth/chonky
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 27d ago · cactuscompute.com
Launched alongside, October 2025
the whole month →
- SA
I went down the rabbit hole on a side project and ended up building this: Strange Attractors(https://blog.shashanktomar.com/posts/strange-attractors). It’s built with three.js. Working on it reminded me of the little "maths for fun" exercises I used to do while learning programming in early days. Just trying things out, getting fascinated and geeky, and being surprised by the results. I spent way too much time on this, but it was extreme fun. My favorite part: someone pointed me to the Simone Attractor on Threads. It is a 2D attractor and I asked GPT to extrapolate it to…
AI · Oct 2025 · blog.shashanktomar.com
- ASAutism Simulator▲779
Hey all, I built this. It’s not trying to capture every autistic experience (that’d be impossible). It’s based on my own lived experience as well as that of friends on the spectrum. I'm trying to give people a feel for what masking, decision fatigue, and burnout can look like day-to-day. That’s hard to explain in words, but easier to show through choices and stats. I'm not trying to "define autism". I’ve gotten good feedback here about resilience, meds, and difficulty tuning. I’ll keep tweaking it. If even a few people walk away thinking, "ah, maybe that’s why my coworker struggles in those…
Life & fun · Oct 2025 · autism-simulator.vercel.app
