Fuzzy deduplicate any CSV using vector embeddings
I made an app to fuzzy-deduplicate my Google Sheets and CRM records - No manual configuration required - Works out-of-the-box on most data types (ex. people, companies, product catalog) Implementation details: - Embeds records using an E5-family model - Performs similarity search using DuckDB w/ vector similarity extension - Does last-mile comparison and merges duplicates using Claude Demo video: https://youtu.be/7mZ0kdwXBwM Github repo (Apache 2.0 licensed): https://github.com/SnowPilotOrg/dedupe_it Background story: My company has a table for…
What it does
In the maker’s words, at launch
I made an app to fuzzy-deduplicate my Google Sheets and CRM records - No manual configuration required - Works out-of-the-box on most data types (ex. people, companies, product catalog) Implementation details: - Embeds records using an E5-family model - Performs similarity search using DuckDB w/ vector similarity extension - Does last-mile comparison and merges duplicates using Claude Demo video: https://youtu.be/7mZ0kdwXBwM Github repo (Apache 2.0 licensed): https://github.com/SnowPilotOrg/dedupe_it Background story: My company has a table for tracking leads, which includes website visitors, demo form submissions, app signups, and manual entries. It’s full of duplicates. And writing formulas to merge those dupes has been a massive PITA. I figured that an LLM could handle any data shape and give me a way to deal with tricky custom rules like “treat international subsidiaries as distinct from their parent company”. The challenging thing was avoiding an NxN comparison matrix. The solution I came up with was first narrowing down our search space using vector embeddings + semantic similarity search, and then using a generative LLM only to compare a few nearest neighbors and merge. Some cool attributes of this approach: - Can work incrementally (no reprocessing the entire dataset) - Allows processing all records in parallel - Composes with deterministic dedupe rules Lmk any feedback on how to make this better!
Does the same job
all alternatives →- GSGoogle Sheets add-on to compare text, fuzzy-match, highlight duplicates2020 · ▲80
I created an add-on for Google Sheets called Flookup, and it comes both as a free version and a VERY AFFORDABLE paid version. At its core, Flookup is a fuzzy matching add-on that helps you manage text that is less than a 100% match. Beyond that it can be used to: 1. Search for and match data regardless of whether it contains typos. 2. Highlight and delete duplicates duplicates even if the data has mismatched text. 3. Calculate the percentage similarity between strings. 4. Extract unique values from any column based on percentage similarity. 5. Sum and find the average of numbers based on…
- SFSemHash – Fast Semantic Text Deduplication for Cleaner Datasets2025 · github.com · ▲6
We’ve just open-sourced SemHash, a lightweight package for semantic text deduplication. It lets you effortlessly clean up your datasets and avoid pitfalls caused by duplicate samples in semantic search, RAG, and machine learning. Main Features: - Fast and hardware friendly: Deduplicate datasets with millions of records in minutes, on a CPU. - Flexible: Works on single or multiple datasets (e.g., train/test deduplication), and multi-column data (e.g., Question-Answering datasets). - Lightweight: Minimal dependencies (largest is NumPy). - Explainable: Easily inspect duplicates and what…
- ALA labelling tool to easily extract and label Wikipedia data2021 · ▲116
Hi HN! I am Maria, solo founder of DataQA (https://dataqa.ai/), a tool to search and label documents for various NLP tasks (e.g. entity extraction, entity linking, etc). I have worked as a data scientist and ML engineer for the better part of a decade, and over that time have specialised mainly in applications involving natural language processing (NLP). One of the key questions I have always had at the back of my mind is whether my time was well spent. Whenever I spent more time on feature engineering or trying different models, I always wondered whether I would get better…
- SSSemHash – Semantic Text Deduplication, Outlier Filtering and Sampling2025 · github.com · ▲7
We’ve just released SemHash v0.3.0, a major rework of our open-source text pre-processing library. We’ve added two new functionalities: outlier filtering & representative sampling. The core API has been reworked to make sure all of these features can be used together in an intuitive way. Our new features use the existing approximate nearest neighbors index that we already used for semantic deduplication, so they can be ran very quickly after building the index on your dataset. The core package can now be used for: - Semantic Deduplication: Remove semantic duplicates from your dataset. This…
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 1d ago · opentrailpaper.com
Launched alongside, November 2024
the whole month →
- IB
I wasn't quite sure if this qualified as "Show HN" given you can't really download it and try it out. However, dang said[0]: > If it's hardware or something that's not so easy to try out over the internet, find a different way to show how it actually works—a video, for example, or a detailed post with photos. Hopefully I did that? Additionally, I've put code and a detailed guide for the netboot computer management setup on GitHub: https://github.com/kentonv/lanparty Anyway, if this shouldn't have been Show HN, I apologize! [0]…
Dev tools · 2024 · lanparty.house
