SemHash – Semantic Text Deduplication, Outlier Filtering and Sampling
We’ve just released SemHash v0.3.0, a major rework of our open-source text pre-processing library. We’ve added two new functionalities: outlier filtering & representative sampling. The core API has been reworked to make sure all of these features can be used together in an intuitive way. Our new features use the existing approximate nearest neighbors index that we already used for semantic deduplication, so they can be ran very quickly after building the index on your dataset. The core package can now be used for: - Semantic Deduplication: Remove semantic duplicates from your dataset. This…
What it does
In the maker’s words, at launch
We’ve just released SemHash v0.3.0, a major rework of our open-source text pre-processing library. We’ve added two new functionalities: outlier filtering & representative sampling. The core API has been reworked to make sure all of these features can be used together in an intuitive way. Our new features use the existing approximate nearest neighbors index that we already used for semantic deduplication, so they can be ran very quickly after building the index on your dataset. The core package can now be used for: - Semantic Deduplication: Remove semantic duplicates from your dataset. This can prevent train/test set overlap in classification tasks, or prevent duplicate samples in RAG/semantic search. - Outlier Filtering: Surface and filter the most anomalous samples from your dataset. This can help with automated removal of low quality data, or data that should not be in your dataset. - Representative Sampling: Select the most central and diverse examples using Maximal Marginal Relevance. This can help you quickly explore and understand a dataset, or even build a small, diverse, high quality dataset, for example for LLM finetuning. We’ve designed these features in the same way as our semantic deduplication: CPU friendly, lightweight, and explainable. We hope these features help you create cleaner datasets, or simply understand your data better. We’re curious to hear your feedback, and whether there are any other features you think would improve SemHash further!
Does the same job
all alternatives →- SFSemHash – Fast Semantic Text Deduplication for Cleaner Datasets2025 · github.com · ▲6
We’ve just open-sourced SemHash, a lightweight package for semantic text deduplication. It lets you effortlessly clean up your datasets and avoid pitfalls caused by duplicate samples in semantic search, RAG, and machine learning. Main Features: - Fast and hardware friendly: Deduplicate datasets with millions of records in minutes, on a CPU. - Flexible: Works on single or multiple datasets (e.g., train/test deduplication), and multi-column data (e.g., Question-Answering datasets). - Lightweight: Minimal dependencies (largest is NumPy). - Explainable: Easily inspect duplicates and what…
- ALA labelling tool to easily extract and label Wikipedia data2021 · ▲116
Hi HN! I am Maria, solo founder of DataQA (https://dataqa.ai/), a tool to search and label documents for various NLP tasks (e.g. entity extraction, entity linking, etc). I have worked as a data scientist and ML engineer for the better part of a decade, and over that time have specialised mainly in applications involving natural language processing (NLP). One of the key questions I have always had at the back of my mind is whether my time was well spent. Whenever I spent more time on feature engineering or trying different models, I always wondered whether I would get better…
- ISI started a repo for sharing algorithm implementations2013 · github.com · ▲52
Everything that would be here is in the README. I hope this gets big, it has tons of potential.
- SBSemantic Bible Search using OpenAI's latest embedding model2023 · siliconscripture.org · ▲8
Hey HN! My brothers and I have worked on this for the last 2 weeks. We use OpenAI's `text-embedding-ada-002` model to embed queries and a vector database to search for similar verses / blocks of verses. We'd like to see what you think and appreciate any feedback!
- SFSemble – Fast code search for agents with near-transformer accuracyApr 2026 · github.com · ▲7
Hey HN! We've just open-sourced Semble, a fast and accurate code search library built for agents. We're also releasing potion-code-16M, a small code-specialized static embedding model that powers it. Most embedding-based code search methods are either too slow to index on demand or need GPU infrastructure, while grep-style retrieval methods often cannot find the relevant content. Semble combines the speed and quality benefits of both, so agents waste less time and fewer tokens exploring. Main features: - Fast: indexes a full codebase in ~250 ms and answers queries in ~1.5 ms, all on CPU…
- UDUsing DSPy to enrich a dataset of the Nobel laureate network2025 · blog.kuzudb.com · ▲8
I've been working a fair bit with DSPy lately, and I did some work in combining the benefits of vector search and LLMs (via a DSPy pipeline) to disambiguate records with a high degree of accuracy to help enrich a dataset. The blog post shows how this approach scales well, is very cost-effective and super concise - all it takes is < 100 lines of DSPy code and it all runs async. The code to reproduce is in this repo if anyone's interested (all tools are 100% free and open source, and the methodology will work with open weight LLMs too).…
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 1d ago · opentrailpaper.com
Launched alongside, April 2025
the whole month →- IB
Hi everyone, I built PyXL — a hardware processor that executes a custom assembly generated from Python programs, without using a traditional interpreter or virtual machine. It compiles Python -> CPython Bytecode -> Instruction set designed for direct hardware execution. I’m sharing an early benchmark: a GPIO test where PyXL achieves a 480ns round-trip toggle — compared to 14-25 micro seconds on a MicroPython Pyboard - even though PyXL runs at a lower clock (100MHz vs. 168MHz). The design is stack-based, fully pipelined, and preserves Python's dynamic typing without static type restrictions.…
Dev tools · 2025 · runpyxl.com
- UC
Life & fun · 2025 · filiph.github.io
- IB
https://the-pocket.github.io/Tutorial-Codebase-Knowledge/
AI · 2025 · github.com
