Life & fun · alternatives · 2026
24 alternatives to PDD – Probabilistic De-Duplication of Streams with Bloom Filters
Advanced Bloom Filter Based Algorithms for Efficient Approximate Data De-Duplication in Streams - jparkie/PDD
PDD is a deduplication tool that uses Bloom filters to identify and remove duplicate entries from data streams. It's designed for developers and data engineers who need to process large volumes of streaming data… Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes. PDD – Probabilistic De-Duplication of Streams with Bloom Filters launched in 2016; newer entries below may have overtaken it.
- 1
😎 Finding duplicate images made easy! Contribute to idealo/imagededup development by creating an account on GitHub.
2019 · github.com · its alternatives →
- 2
DeDuplicate▲78Remove duplicate files in G Drive, Dropbox, OneDrive, MEGA
2022 · apps.apple.com · its alternatives →
- 3
- 4PS
Hello HN, I am Sai Srirampur, one of the Co-founders of PeerDB. (https://github.com/PeerDB-io/peerdb). We spent the past 7 months building a solid experience to replicate data from Postgres to data warehouses. Now we're expanding to queues. PeerDB Streams provides a simple and native way to replicate changes as they happen in Postgres to Queues (Kafka, Redpanda, Google PubSub, etc). We use Postgres logical decoding to enable Change Data Capture (CDC). Blog post here: https://blog.peerdb.io/peerdb-streams-simple-native-postgres.... 10-min quickstart here:…
2024 · its alternatives →
- 5
Embedding-based code duplication detector. Contribute to rafal-qa/slopo development by creating an account on GitHub.
Jul 2026 · github.com · its alternatives →
- 6
Cuckoo Filter implementation in Go, better than Bloom Filters (unmaintained) - irfansharif/cfilter
2016 · github.com · its alternatives →
- 7
:crystal_ball: Scalable reverse image search built on Kubernetes and Elasticsearch - dsys/match
2016 · github.com · its alternatives →
- 8
tl;dr we built an embeddable stream processing engine in Rust using apache DataFusion, check us out at https://github.com/probably-nothing-labs/denormalized Hey HN, We’d like to showcase a very early version of our embeddable stream processing engine called Denormalized. The rise of DuckDB has abundantly made it clear that even for many workloads of Terabyte scale, a single node system outshines the distributed query engines of previous generation such as Spark, Snowflake etc in terms of both performance and cost. Now a lot of workloads DuckDB is used for were normally…
2024 · github.com · its alternatives →
- 9
Probabilistic data structures for processing continuous, unbounded streams. - tylertreat/BoomFilters
2015 · github.com · its alternatives →
- 10
Hi HN! We are Ashish and Armend, founders of GlassFlow. We just launched our open-source streaming ETL that deduplicates and joins Kafka streams before ingesting them to ClickHouse https://github.com/glassflow/clickhouse-etl Why we built this: Dedup with batch data is straightforward. You load the data into a temporary table. Then, find only the latest versions of the record through hashes or keys and keep them. After that, move the clean data into your main table. But have you tried this with streaming data? Users of our prev product were running real-time analytics…
2025 · github.com · its alternatives →
- 11
Distributed Stream and Batch Processing. Contribute to hazelcast/hazelcast-jet development by creating an account on GitHub.
2020 · github.com · its alternatives →
- 12
We’ve just open-sourced SemHash, a lightweight package for semantic text deduplication. It lets you effortlessly clean up your datasets and avoid pitfalls caused by duplicate samples in semantic search, RAG, and machine learning. Main Features: - Fast and hardware friendly: Deduplicate datasets with millions of records in minutes, on a CPU. - Flexible: Works on single or multiple datasets (e.g., train/test deduplication), and multi-column data (e.g., Question-Answering datasets). - Lightweight: Minimal dependencies (largest is NumPy). - Explainable: Easily inspect duplicates and what…
2025 · github.com · its alternatives →
- 13
High-Performance GPU Cuckoo Filter. Contribute to tdortman/Cuckoo-GPU development by creating an account on GitHub.
Jan 2026 · github.com · its alternatives →
- 14
All-pair set similarity search on millions of sets in Python and on a laptop - GitHub - ekzhu/SetSimilaritySearch: All-pair set similarity search on millions of sets in Python and on a laptop
2018 · github.com · its alternatives →
- 15DD
Gleb, Alex, Erez and Simon here – we are building an open-source tool for comparing data within and across databases at any scale. The repo is at https://github.com/datafold/data-diff, and our home page is https://datafold.com/. As a company, Datafold builds tools for data engineers to automate the most tedious and error-prone tasks falling through the cracks of the modern data stack, such as data testing and lineage. We launched two years ago with a tool for regression-testing changes to ETL code…
2022 · its alternatives →
- 16
Deduplix▲9Deduplication Software and Secure Data Deduplication
2025 · ixsight.com · its alternatives →
- 17
MemStream: Memory-Based Streaming Anomaly Detection - Stream-AD/MemStream
2021 · github.com · its alternatives →
- 18
Cuckoo Filter: Practically better than bloom filter - vedhavyas/cuckoo-filter
2017 · github.com · its alternatives →
- 19
Contribute to seiflotfy/pcsa development by creating an account on GitHub.
2018 · github.com · its alternatives →
- 20
DedupX▲5Hi HN, I built a duplicate file finder for my brother who's a photography enthusiast and constantly runs out of storage constantly due to resizing a lot of photos and having a lot of duplicates around. Notes: - Incremental hashing: Instead of loading entire files into memory, I hash files in chunks. Files with identical sizes get grouped and progressively hashed until they diverge or match completely. - Perceptual hashing: For images, I use perceptual hashing (pHash) that generates a fingerprint based on visual content rather than bytes. Similar images have similar hashes. - BK-Tree…
Oct 2025 · maheepk.net · its alternatives →
- 21
A compressed, sparse cuckoo filter (see https://www.vldb.org/pvldb/vol11/p1041-breslow.pdf) - AMDComputeLibraries/morton_filter
2019 · github.com · its alternatives →
- 22
Hi! We are excited to announce the second release of Desbordante — an open-source, high-performance data profiler that is capable of discovering and validating many different patterns in data using various algorithms. Unlike existing data profilers, Desbordante focuses on discovering complex patterns in data, which are notoriously hard to extract. Since its inception in 2019, it has become the fastest open-source tool for these tasks. It also offers an array of patterns which have no alternative implementations. With this release, Desbordante now supports 17 types of patterns, such as:…
2024 · github.com · its alternatives →
- 23BFBloom Filter▲65
2017 · sacert.github.io · its alternatives →
- 24
Automatically concurrent data fetching and request deduplication in C#. - joashc/HaxlSharp
2016 · github.com · its alternatives →
Also compare
- Imagededup – Finding duplicate images made easy alternatives
- DeDuplicate alternatives
- DepsHub alternatives
- Peerdb Streams – Simple, native Postgres change data capture alternatives
- CLI tool for detecting non-exact code duplication with embedding models alternatives
- Cuckoo Filter Implementation in Go, Better Than Bloom Filters alternatives
Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →