Alternatives
Products that do what SemHash – Fast Semantic Text Deduplication for Cleaner Datasets does
We’ve just open-sourced SemHash, a lightweight package for semantic text deduplication. It lets you effortlessly clean up your datasets and avoid pitfalls caused by duplicate samples in semantic search, RAG, and machine learning. Main Features: - Fast and hardware friendly: Deduplicate datasets with millions of records in minutes, on a CPU. - Flexible: Works on single or multiple datasets (e.g., train/test deduplication), and multi-column data (e.g., Question-Answering datasets). - Lightweight: Minimal dependencies (largest is NumPy). - Explainable: Easily inspect duplicates and what…
- 1SS
We’ve just released SemHash v0.3.0, a major rework of our open-source text pre-processing library. We’ve added two new functionalities: outlier filtering & representative sampling. The core API has been reworked to make sure all of these features can be used together in an intuitive way. Our new features use the existing approximate nearest neighbors index that we already used for semantic deduplication, so they can be ran very quickly after building the index on your dataset. The core package can now be used for: - Semantic Deduplication: Remove semantic duplicates from your dataset. This…
2025 · github.com
- 2AL
Hi HN! I am Maria, solo founder of DataQA (https://dataqa.ai/), a tool to search and label documents for various NLP tasks (e.g. entity extraction, entity linking, etc). I have worked as a data scientist and ML engineer for the better part of a decade, and over that time have specialised mainly in applications involving natural language processing (NLP). One of the key questions I have always had at the back of my mind is whether my time was well spent. Whenever I spent more time on feature engineering or trying different models, I always wondered whether I would get better…
2021
- 3PA
2017 · github.com
- 4OA
2016 · github.com
- 5DC
2016 · datasets.co
- 6DF
2020 · github.com
- 7TS
2021 · mro.name
- 8LA
2017 · github.com
- 9AR
2015 · github.com
- 10PG
2017 · github.com
- 11SF
Hey HN! We've just open-sourced Semble, a fast and accurate code search library built for agents. We're also releasing potion-code-16M, a small code-specialized static embedding model that powers it. Most embedding-based code search methods are either too slow to index on demand or need GPU infrastructure, while grep-style retrieval methods often cannot find the relevant content. Semble combines the speed and quality benefits of both, so agents waste less time and fewer tokens exploring. Main features: - Fast: indexes a full codebase in ~250 ms and answers queries in ~1.5 ms, all on CPU…
Apr 2026 · github.com
- 12AG
2017 · github.com
- 13FD
I made an app to fuzzy-deduplicate my Google Sheets and CRM records - No manual configuration required - Works out-of-the-box on most data types (ex. people, companies, product catalog) Implementation details: - Embeds records using an E5-family model - Performs similarity search using DuckDB w/ vector similarity extension - Does last-mile comparison and merges duplicates using Claude Demo video: https://youtu.be/7mZ0kdwXBwM Github repo (Apache 2.0 licensed): https://github.com/SnowPilotOrg/dedupe_it Background story: My company has a table for…
2024 · app.dedupe.it
- 14MT
2015 · monkeylearn.com
- 15UD
I've been working a fair bit with DSPy lately, and I did some work in combining the benefits of vector search and LLMs (via a DSPy pipeline) to disambiguate records with a high degree of accuracy to help enrich a dataset. The blog post shows how this approach scales well, is very cost-effective and super concise - all it takes is < 100 lines of DSPy code and it all runs async. The code to reproduce is in this repo if anyone's interested (all tools are 100% free and open source, and the methodology will work with open weight LLMs too).…
2025 · blog.kuzudb.com
- 16KA
2019 · github.com
- 17GC
2021 · gist.github.com
- 18RL
2023 · app.refuel.ai
- 19AA
2020 · github.com
- 20SB
Hey HN! My brothers and I have worked on this for the last 2 weeks. We use OpenAI's `text-embedding-ada-002` model to embed queries and a vector database to search for similar verses / blocks of verses. We'd like to see what you think and appreciate any feedback!
2023 · siliconscripture.org
- 21MM
Hi HN! We (Thomas and Stéphan, hello!) recently released Model2Vec, a Python library for distilling any sentence transformer into a small set of static embeddings. This makes inference with such a model up to 500x faster, and reduces model size by a factor of 15 (7.5M params or 15/30MB on disk, depending on whether you use float16 or float32). This allows you to embed 50-100k documents per second on a cpu on a macbook. This reduction of course comes at a cost: distilled models are worse than their parent models. Even so, they are actually a lot better than large sets of conventional…
2024 · github.com
- 22AE
Hey folks, Elias here. Excited to unveil my latest project. Why I Built This: Traditional keyword search isn't cutting it. I've used LLM-embeddings to provide more nuanced, relevant results. How It Works: LLM-embedding similarity on curated datasets for semantically similar results. No need to iterate over keywords any more. Current Datasets: - YC Companies - Show HN Posts, - Ask HN Posts - ProductHunt Startups - Github Top 200k Repos Use Cases: - Validate a product idea's existence - Check if someone already Asked HN something - Have fun - search random terms and see what pops up Want to…
2023 · payperrun.com
- 23H4
2018 · usenix.org
- 24TS
2020 · textpiper.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →