AI · alternatives · 2026
24 alternatives to SemHash – Fast Semantic Text Deduplication for Cleaner Datasets
We’ve just open-sourced SemHash, a lightweight package for semantic text deduplication. It lets you effortlessly clean up your datasets and avoid pitfalls caused by duplicate samples in semantic search, RAG, and machine learning. Main Features: - Fast and hardware friendly: Deduplicate datasets with millions of records in minutes, on a CPU. - Flexible: Works on single or multiple datasets (e.g., train/test deduplication), and multi-column data (e.g., Question-Answering datasets). - Lightweight: Minimal dependencies (largest is NumPy). - Explainable: Easily inspect duplicates and what…
SemHash is an open-source Python package for identifying and removing semantically similar text from datasets. It deduplicates millions of records quickly on standard CPUs with minimal dependencies, making it useful… Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes.
- 1
After working with LLMs for long enough, I found myself wanting a lightweight utility for doing various small tasks to prepare inputs, locate information and create evaluators. This library is two things: a very simple model and utilities that inference it (eg. fuzzy deduplication). The target platform is CPU, and it’s intended to be light, fast and pip installable — a library that lowers the barrier to working with strings semantically. You don’t need to install pytorch to use it, or any deep learning runtimes. How can this be accomplished? The model is simply token embeddings that are…
2024 · github.com · its alternatives →
- 2
Much improved new version. Search for words similar to the query. For example, "death" will find "death", "dying", "dead", "killing"... Incredibly useful for exploring large text datasets where exact matches are too restrictive.
2024 · github.com · its alternatives →
- 3
Hey HN! We (Stephan and Thomas) recently open-sourced Semble. We kept running into the same problem while using Claude Code on large codebases: when the agent can't find something directly, it falls back to grep, reading full files or launching subagents. This uses a lot of tokens, and often still misses the relevant code. There are existing tools for this, but they were either too slow to index on demand, needed API keys, or had poor retrieval quality. Semble is our solution for this. It combines static Model2Vec embeddings (using our latest static model: potion-code-16M) with BM25, fused…
May 2026 · github.com · its alternatives →
- 4
TLDR: I’ve made a transformer model and a wrapper library that segments text into meaningful semantic chunks. The current text splitting approaches rely on heuristics (although one can use neural embedder to group semantically related sentences). I propose a fully neural approach to semantic chunking. I took the base distilbert model and trained it on a bookcorpus to split concatenated text paragraphs into original paragraphs. Basically it’s a token classification task. Model fine-tuning took day and a half on a 2x1080ti. The library could be used as a text splitter module in a RAG system or…
2025 · github.com · its alternatives →
- 5
All-pair set similarity search on millions of sets in Python and on a laptop - GitHub - ekzhu/SetSimilaritySearch: All-pair set similarity search on millions of sets in Python and on a laptop
2018 · github.com · its alternatives →
- 6
Local first semantic and hybrid BM25 grep / search tool for use by AI and humans! - GitHub - BeaconBay/ck: Local first semantic and hybrid BM25 grep / search tool for use by AI and humans!
2025 · github.com · its alternatives →
- 7
Advanced Bloom Filter Based Algorithms for Efficient Approximate Data De-Duplication in Streams - jparkie/PDD
2016 · github.com · its alternatives →
- 8IM
As a grad student (and an ADHDer), I had trouble doing literature review systematically. To combat this, I made a website that finds similar papers using the meaning of the thing I am looking for. I used MixedBread's [^1] embedding model to generate vectors from the abstracts. I store and search similar vectors using Milvus [^2] and finally use Gradio [^3] to serve the frontend. I update the vector database weekly by pulling the metadata dataset from Kaggle [^4]. To speed up the search process on my free oracle instance, I binarise the embeddings and use Hamming distance as a metric. I would…
2024 · papermatch.mitanshu.tech · its alternatives →
- 95N
2021 · thishashcollisionisnotporn.com · its alternatives →
- 10
I built Chonkie because I was tired of rewriting chunking code for RAG applications. Existing libraries were either too bloated (80MB+) or too basic, with no middle ground. Core features: - 21MB default install vs 80-171MB alternatives - 33x faster token chunking than popular alternatives - Supports multiple chunking strategies: token, word, sentence, and semantic - Works with all major tokenizers (transformers, tokenizers, tiktoken) - Zero external dependencies for basic functionality Technical optimizations: - Uses tiktoken with multi-threading for faster tokenization - Implements…
2024 · github.com · its alternatives →
- 11
Build data processing and data analysis pipelines that leverage the power of LLMs 🧠 - anishathalye/semlib
2025 · github.com · its alternatives →
- 12
I recently found myself computing the similarity between lots of very high dimensional vectors (i.e., sentence embedding vectors from LLMs), and I wanted to try some more powerful measures of similarity/dependency than just Cosine similarity, which seems to be the default for everything nowadays because of its computational efficiency. There are many other more involved measures that can detect more subtle relationships, but the problem is that some of them are quite slow to compute, especially if you're trying to do it in Python. For my favorite measure of statistical dependency,…
2023 · github.com · its alternatives →
- 13
Introducing Biblos, a simple tool for semantic search and summarization of Bible passages. Leveraging Chroma for vector search with BAAI BGE embeddings, semantically find related verses across the Bible. The tool employs Anthropic's Claude LLM model for generating high-quality summaries of retrieved passages, contextualizing your search topic. Built on a Retrieval Augmented Generation (RAG) architecture, the app implements a simple Streamlit Web UI using Python. Deployed using render.com, the app is available at https://biblos.app Note: Search by just topic/keywords, e.g.…
2023 · github.com · its alternatives →
- 14
I've built an advanced RAG (Retrieval-Augmented Generation) pipeline from scratch to demystify the complex mechanics of modern LLM-powered Question Answering systems. This repository features: -- An implementation of a sub-question query engine from scratch to answer complex user questions. -- Illustrative explanations that unveil the inner workings of the system. -- An analysis of the challenges I faced while working with the system, like prompt engineering and cost estimation. -- Qualitative comparison with similar frameworks like LlamaIndex, offering a broader perspective. Key Takeaway:…
2023 · github.com · its alternatives →
- 15AL
Hi HN! I am Maria, solo founder of DataQA (https://dataqa.ai/), a tool to search and label documents for various NLP tasks (e.g. entity extraction, entity linking, etc). I have worked as a data scientist and ML engineer for the better part of a decade, and over that time have specialised mainly in applications involving natural language processing (NLP). One of the key questions I have always had at the back of my mind is whether my time was well spent. Whenever I spent more time on feature engineering or trying different models, I always wondered whether I would get better…
2021 · its alternatives →
- 16
HyperMinHash: Bringing intersections to HyperLogLog - axiomhq/hyperminhash
2018 · github.com · its alternatives →
- 17
:crystal_ball: Scalable reverse image search built on Kubernetes and Elasticsearch - dsys/match
2016 · github.com · its alternatives →
- 18
We’ve recently open-sourced Model2vec, a method to distill sentence transformers into static embeddings that outperform all previous approaches by a large margin on MTEB. Our new models set a new state-of-the-art for static embeddings. Main features: - Our best model (potion-base-8M) has only 8M parameters, which is ~30mb on disk - Inference is ~500x faster than the distilled base model (bge-base), on a CPU - New models can be distilled in 30 seconds on a CPU without requiring a dataset - just a vocabulary - Numpy-only inference: The packaged can be install the package with minimal…
2024 · github.com · its alternatives →
- 19
Embedding-based code duplication detector. Contribute to rafal-qa/slopo development by creating an account on GitHub.
Jul 2026 · github.com · its alternatives →
- 20
Hi HN, I built SplatHash. It's a lightweight image placeholder generator I wrote to be a simpler, faster alternative to BlurHash and ThumbHash. Repo: https://github.com/junevm/splathash
Feb 2026 · github.com · its alternatives →
- 21
TLDR: I’m expanding the family of text-splitting Chonky models with new multilingual model. You can learn more about this neural approach in a previous post: https://news.ycombinator.com/item?id=43652968 Since the release of the first distilbert-based model I’ve released two more models based on a ModernBERT. All these models were pre-trained and fine-tuned primary on English texts. But recently mmBERT(https://huggingface.co/blog/mmbert) has been released. This model pre-trained on massive dataset that contains 1833 languages. So I had an idea of…
Oct 2025 · huggingface.co · its alternatives →
- 22
I have a proposal that addresses long-term memory problems for LLMs when new data arrives continuously (cheaply!). The program involves no code, but two Markdown files. For retrieval, there is a semantic filesystem that makes it easy for LLMs to search using shell commands. It is currently a scrappy v1, but it works better than anything I have tried. Curious for any feedback!
Apr 2026 · github.com · its alternatives →
- 23
A simple re-implementation of Google's sparsehash as a learning excercise. - qpfiffer/Simple-Sparsehash
2015 · github.com · its alternatives →
- 24UD
I've been working a fair bit with DSPy lately, and I did some work in combining the benefits of vector search and LLMs (via a DSPy pipeline) to disambiguate records with a high degree of accuracy to help enrich a dataset. The blog post shows how this approach scales well, is very cost-effective and super concise - all it takes is < 100 lines of DSPy code and it all runs async. The code to reproduce is in this repo if anyone's interested (all tools are 100% free and open source, and the methodology will work with open weight LLMs too).…
2025 · blog.kuzudb.com · its alternatives →
Also compare
- Wordllama – Things you can do with the token embeddings of an LLM alternatives
- Semantic Grep – A Word2Vec-powered search tool alternatives
- Semble – Code search for agents that uses 98% fewer tokens than grep alternatives
- Chonky – a neural approach for text semantic chunking alternatives
- All-pair similarity search on millions of sets in Python and on laptop alternatives
- Semantic grep with local embeddings alternatives
Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →