SemHash – Fast Semantic Text Deduplication for Cleaner Datasets
We’ve just open-sourced SemHash, a lightweight package for semantic text deduplication. It lets you effortlessly clean up your datasets and avoid pitfalls caused by duplicate samples in semantic search, RAG, and machine learning. Main Features: - Fast and hardware friendly: Deduplicate datasets with millions of records in minutes, on a CPU. - Flexible: Works on single or multiple datasets (e.g., train/test deduplication), and multi-column data (e.g., Question-Answering datasets). - Lightweight: Minimal dependencies (largest is NumPy). - Explainable: Easily inspect duplicates and what…
SemHash – Fast Semantic Text Deduplication for Cleaner Datasets launched on January 12, 2025 with 19 votes, #764 of 6,142 launches that month and more than 90% of that year's launches. The site was still online in September 2026, 1 year on.
In plain words
SemHash is an open-source Python package for identifying and removing semantically similar text from datasets. It deduplicates millions of records quickly on standard CPUs with minimal dependencies, making it useful for cleaning training data in machine learning, semantic search, and retrieval-augmented generation systems. The tool handles single or multiple datasets and multi-column data like question-answer pairs, while allowing users to inspect duplicates and adjust similarity thresholds to suit their needs.
written from the facts on this page · September 2026
From the sources
Fast Multimodal Semantic Deduplication & Filtering - MinishLab/semhash
In the maker’s words, at launch
We’ve just open-sourced SemHash, a lightweight package for semantic text deduplication. It lets you effortlessly clean up your datasets and avoid pitfalls caused by duplicate samples in semantic search, RAG, and machine learning. Main Features: - Fast and hardware friendly: Deduplicate datasets with millions of…
Does a similar job
all alternatives →
Wordllama – Things you can do with the token embeddings of an LLM2024 · github.com · ▲370After working with LLMs for long enough, I found myself wanting a lightweight utility for doing various small tasks to prepare inputs, locate information and create evaluators. This library is two things: a very simple model and utilities that inference it (eg. fuzzy deduplication). The target platform is CPU, and it’s intended to be light, fast and pip installable — a library that lowers the barrier to working with strings semantically. You don’t need to install pytorch to use it, or any deep learning runtimes. How can this be accomplished? The model is simply token embeddings that are…
Semantic Grep – A Word2Vec-powered search tool2024 · github.com · ▲356Much improved new version. Search for words similar to the query. For example, "death" will find "death", "dying", "dead", "killing"... Incredibly useful for exploring large text datasets where exact matches are too restrictive.
Semble – Code search for agents that uses 98% fewer tokens than grepMay 2026 · github.com · ▲445Hey HN! We (Stephan and Thomas) recently open-sourced Semble. We kept running into the same problem while using Claude Code on large codebases: when the agent can't find something directly, it falls back to grep, reading full files or launching subagents. This uses a lot of tokens, and often still misses the relevant code. There are existing tools for this, but they were either too slow to index on demand, needed API keys, or had poor retrieval quality. Semble is our solution for this. It combines static Model2Vec embeddings (using our latest static model: potion-code-16M) with BM25, fused…
Chonky – a neural approach for text semantic chunking2025 · github.com · ▲169TLDR: I’ve made a transformer model and a wrapper library that segments text into meaningful semantic chunks. The current text splitting approaches rely on heuristics (although one can use neural embedder to group semantically related sentences). I propose a fully neural approach to semantic chunking. I took the base distilbert model and trained it on a bookcorpus to split concatenated text paragraphs into original paragraphs. Basically it’s a token classification task. Model fine-tuning took day and a half on a 2x1080ti. The library could be used as a text splitter module in a RAG system or…
All-pair similarity search on millions of sets in Python and on laptop2018 · github.com · ▲123All-pair set similarity search on millions of sets in Python and on a laptop - GitHub - ekzhu/SetSimilaritySearch: All-pair set similarity search on millions of sets in Python and on a laptop
Semantic grep with local embeddings2025 · github.com · ▲179Local first semantic and hybrid BM25 grep / search tool for use by AI and humans! - GitHub - BeaconBay/ck: Local first semantic and hybrid BM25 grep / search tool for use by AI and humans!
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 21d ago · stillwet.art
FastRouter.ai▲364Route requests to the right LLM for cost, latency & quality
AI · 22d ago · fastrouter.ai
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, January 2025
the whole month →
Hello! I'm Byran. I spent the past ~6 months engineering a laptop from scratch. It's fully open-source on GH at: https://github.com/Hello9999901/laptop
Dev tools · 2025 · byran.ee
- TITetris in a PDF▲1,289
I realized that the PDF engines of modern desktop browsers (PDFium and PDF.js) support JavaScript with enough I/O primitives to make a basic game like Tetris. It was a bit tricky to find a union of features that work in both engines, but in the end it turns out that showing/hiding annotation "fields" works well to make monochrome pixels, and keyboard input can be achieved by typing in a text input box. All in all it's quite janky but a nice reminder of how general purpose PDF scripting can be. The linked PDF is all ASCII so you can just open it in a text editor, or have a look at…
Life & fun · 2025 · th0mas.nl
21st.dev▲1,211I originally built this app to organize and reuse interesting React + Tailwind components I came across on Twitter, inspired by the shadcn/ui approach. Later, a friend started contributing their own components, and it got me thinking—this could grow into something bigger, like an NPM for design engineers to share and use components. Would love to hear your thoughts!
Dev tools · 2025 · 21st.dev
Create lifelike, personalized AI avatars from text prompts
AI · 2025 · jogg.ai
CSS library based on Counter Strike 1.6 UI.
Life & fun · 2025 · cs16.samke.me