Alternatives

Products that do what Chonky – a neural approach for text semantic chunking does

TLDR: I’ve made a transformer model and a wrapper library that segments text into meaningful semantic chunks. The current text splitting approaches rely on heuristics (although one can use neural embedder to group semantically related sentences). I propose a fully neural approach to semantic chunking. I took the base distilbert model and trained it on a bookcorpus to split concatenated text paragraphs into original paragraphs. Basically it’s a token classification task. Model fine-tuning took day and a half on a 2x1080ti. The library could be used as a text splitter module in a RAG system or…

1. Not indexed yet — check back in a few minutes.

Ranked by how close each launch is in meaning, then by votes. Refine with a description →