Alternatives
Products that do what Chonky – a neural text semantic chunking goes multilingual does
TLDR: I’m expanding the family of text-splitting Chonky models with new multilingual model. You can learn more about this neural approach in a previous post: https://news.ycombinator.com/item?id=43652968 Since the release of the first distilbert-based model I’ve released two more models based on a ModernBERT. All these models were pre-trained and fine-tuned primary on English texts. But recently mmBERT(https://huggingface.co/blog/mmbert) has been released. This model pre-trained on massive dataset that contains 1833 languages. So I had an idea of…
- 1CA
TLDR: I’ve made a transformer model and a wrapper library that segments text into meaningful semantic chunks. The current text splitting approaches rely on heuristics (although one can use neural embedder to group semantically related sentences). I propose a fully neural approach to semantic chunking. I took the base distilbert model and trained it on a bookcorpus to split concatenated text paragraphs into original paragraphs. Basically it’s a token classification task. Model fine-tuning took day and a half on a 2x1080ti. The library could be used as a text splitter module in a RAG system or…
2025 · github.com
- 2CA
I built Chonkie because I was tired of rewriting chunking code for RAG applications. Existing libraries were either too bloated (80MB+) or too basic, with no middle ground. Core features: - 21MB default install vs 80-171MB alternatives - 33x faster token chunking than popular alternatives - Supports multiple chunking strategies: token, word, sentence, and semantic - Works with all major tokenizers (transformers, tokenizers, tiktoken) - Zero external dependencies for basic functionality Technical optimizations: - Uses tiktoken with multi-threading for faster tokenization - Implements…
2024 · github.com
- 3IM
I built this project as a way to learn more about NLP by applying it to something weird and unsolved. The Voynich Manuscript is a 15th-century book written in an unknown script. No one’s been able to translate it, and many think it’s a hoax, a cipher, or a constructed language. I wasn’t trying to decode it — I just wanted to see: does it behave like a structured language? I stripped a handful of common suffix-like endings (aiin, dy, etc.) to isolate what looked like root forms. I know that’s a strong assumption — I call it out directly in the repo — but it helped clarify the clustering. From…
2025 · github.com
- 4
- 5
- 6AC
Hi HN, We’re Shreyash and Bhavnick. We built Chonkie, an open-source library for advanced chunking and embedding of text and code. It was previously Python-only, but we just released a TypeScript version: https://github.com/chonkie-inc/chonkie-ts Many AI projects in JS/TS (like those using Vercel's AI SDK or Mastra) rely on basic text splitters. But better chunking = better retrieval = better performance. That’s what Chonkie is built for. Current native chunkers (in TS): - Code Chunker – handles Python, TypeScript, etc. - Recursive Chunker – rule-based, hierarchical…
2025
- 7KG
Hi HN! My latest side project is knowledge graph that maps the French culinary network using data extracted from restaurant reviews from LeFooding.com. The project uses LLMs to extract structured information from unstructured text. Some technical aspects you may be interested in: - Used structured generation to reliably parse unstructured text into a consistent schema - Tested multiple models (Mistral-7B-v0.3, Llama3.2-3B, gpt4o-mini) for information extraction - Created an interactive visualization using gephi-lite and Retina (WebGL) - Built (with Claude) a simple Flask web app to clean and…
2025 · theophilecantelob.re
- 8
- 9
- 10S8
Feb 2026 · guidelabs.ai
- 11AD
Hey HN! I love watching YouTube with my 7-year-old daughter. Unfortunately, the best stuff is often in English (we're German). So I made an AI tool that translates videos directly, using the original voices. All other sounds, as well as background music, are preserved, too. Turns out that it works for many other language pairs, too. So far, it can create dubs in English, Mandarin Chinese, Spanish, Arabic, French, Russian, German, Italian, Korean, Polish and Dutch. The main challenge in building this was to get the balance right between translating the original meaning and getting the timing…
2024 · speakz.ai
- 12LO
2021 · libretranslate.com
- 13BT
2024 · github.com
- 14
- 15WF
We have a dataset of 3,095 standardized AI responses across 43 prompts. From each response, we extract a 32-dimension stylometric fingerprint (lexical richness, sentence structure, punctuation habits, formatting patterns, discourse markers). Some findings: - 9 clone clusters (>90% cosine similarity on z-normalized feature vectors) - Mistral Large 2 and Large 3 2512 score 84.8% on a composite metric combining 5 independent signals - Gemini 2.5 Flash Lite writes 78% like Claude 3 Opus. Costs 185x less - Meta has the strongest provider "house style" (37.5x distinctiveness ratio) - "Satirical…
Apr 2026 · rival.tips
- 16
- 17TD
This is a character-level language diffusion model for text generation. The model is a modified version of Nanochat's GPT implementation and is trained on Tiny Shakespeare! It is only 10.7 million parameters, so you can try it out locally.
Nov 2025 · github.com
- 18
- 19PL
Hey there HN - Today I'm sharing a demo of my our language learning tool, Phrasing. It's a tool born from the language acquisition hypothesis, too many hours in an anki slog, and a strange desire to always be learning obscure languages. The method is simple: 1. type in a show 2. learn the most important words 3. watch the show/acquire the words 4. review the words when needed in the future. On top of that, we're trying to: - do some novel things with spaced repetition (no more anki slog) - expand the sort of content you can learn from (I want to to refresh my French by reading The…
2024 · phrasing.app
- 20
- 21SC
I've been playing with embeddings and wanted to try out what results the embedding layer will produce based on just word-by-word input and addition / subtraction, beyond what many videos / papers mention (like the obvious king-man+woman=queen). So I built something that doesn't just give the first answer, but ranks the matches based on distance / cosine symmetry. I polished it a bit so that others can try it out, too. For now, I only have nouns (and some proper nouns) in the dataset, and pick the most common interpretation among the homographs. Also, it's case sensitive.
2025 · calc.datova.ai
- 22AE
Hi HN, Stefan here. autofit2 is a project I have been using at my previous company and is now opensourced. It has been used extensively in automated text moderation, but can be applied to any text/document classification task. We had success modeling offensive texts in 20+ languages (cf. github.com/neospe/dataload for all the datasets). It's an integrated pipeline for lightweight multilingual text classification, covering preprocessing, training, and evaluation. It implements SetFit, a few-shot learning technique that works well for low-data regimes (down to a few dozen…
Jun 2026 · github.com
- 23ML
We’ve recently open-sourced Model2vec, a method to distill sentence transformers into static embeddings that outperform all previous approaches by a large margin on MTEB. Our new models set a new state-of-the-art for static embeddings. Main features: - Our best model (potion-base-8M) has only 8M parameters, which is ~30mb on disk - Inference is ~500x faster than the distilled base model (bge-base), on a CPU - New models can be distilled in 30 seconds on a CPU without requiring a dataset - just a vocabulary - Numpy-only inference: The packaged can be install the package with minimal…
2024 · github.com
- 24CT
I had been looking to try <500M parameter language models but you wouldn't find an API to try them anywhere, so I built this cloudflare hosted static website that hosts weights and built an inference runtime for these models that uses WebGPU and runs inference from your browser. These are only so useful in a multi-turn conversation but it's still interesting to see what you can pack in a <250mb model. I tried using ONNX versions earlier, but there were too many quirks of using them with language models and the TPS wasn't too impressive. Inspired by svenflow/webgpu-gemma, I put my codex…
May 2026 · chonklm.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →