PDD – Probabilistic De-Duplication of Streams with Bloom Filters
Advanced Bloom Filter Based Algorithms for Efficient Approximate Data De-Duplication in Streams - jparkie/PDD
PDD – Probabilistic De-Duplication of Streams with Bloom Filters launched on December 5, 2016 with 86 votes, #471 of 3,449 launches that month and more than 95% of that year's launches. The site was still online in September 2026, 9 years on.
In plain words
PDD is a deduplication tool that uses Bloom filters to identify and remove duplicate entries from data streams. It's designed for developers and data engineers who need to process large volumes of streaming data efficiently. The tool leverages probabilistic algorithms to detect duplicates with minimal memory overhead, making it suitable for high-throughput applications where traditional exact-match deduplication would be impractical or resource-intensive.
written from the facts on this page · September 2026
Does a similar job
all alternatives →
Imagededup – Finding duplicate images made easy2019 · github.com · ▲290😎 Finding duplicate images made easy! Contribute to idealo/imagededup development by creating an account on GitHub.
- PSPeerdb Streams – Simple, native Postgres change data capture2024 · ▲177
Hello HN, I am Sai Srirampur, one of the Co-founders of PeerDB. (https://github.com/PeerDB-io/peerdb). We spent the past 7 months building a solid experience to replicate data from Postgres to data warehouses. Now we're expanding to queues. PeerDB Streams provides a simple and native way to replicate changes as they happen in Postgres to Queues (Kafka, Redpanda, Google PubSub, etc). We use Postgres logical decoding to enable Change Data Capture (CDC). Blog post here: https://blog.peerdb.io/peerdb-streams-simple-native-postgres.... 10-min quickstart here:…
CLI tool for detecting non-exact code duplication with embedding modelsJul 2026 · github.com · ▲91Embedding-based code duplication detector. Contribute to rafal-qa/slopo development by creating an account on GitHub.
Scalable reverse image search built on Kubernetes and Elasticsearch2016 · github.com · ▲139:crystal_ball: Scalable reverse image search built on Kubernetes and Elasticsearch - dsys/match
Denormalized – Embeddable Stream Processing in Rust and DataFusion2024 · github.com · ▲125tl;dr we built an embeddable stream processing engine in Rust using apache DataFusion, check us out at https://github.com/probably-nothing-labs/denormalized Hey HN, We’d like to showcase a very early version of our embeddable stream processing engine called Denormalized. The rise of DuckDB has abundantly made it clear that even for many workloads of Terabyte scale, a single node system outshines the distributed query engines of previous generation such as Spark, Snowflake etc in terms of both performance and cost. Now a lot of workloads DuckDB is used for were normally…
More life & fun this month
the category →
A free city-building game in your browser. You are the mayor. You know what the city needs. Now get it past the zoning board, the neighbors, the appeals court and the next election.
Life & fun · 3d ago · permit.petercai.com
- IV
Life & fun · 16d ago · tanzaniaeducationproject.org
Turn a phone, tablet or TV into a sign. Type a message and it fills the screen: welcome signs, timers, Wi-Fi passwords and more. No app, no account, free.
Life & fun · 17d ago · bigwords.page
Play a free virtual guitar online with your keyboard or by touch. Slow songs down and see every chord, string and strum. Acoustic, classic, electric and bass.
Life & fun · 10d ago · guitoro.com
Carrier settings from iPhone, Pixel and Galaxy firmware, decoded and compared: APNs, VoLTE, 5G and Wi-Fi Calling per carrier, and what each build changed.
Life & fun · 14d ago · carrierexplode.com
Life & fun · 24d ago · space.bl2.net
Source: Show HN post ↗
Launched alongside, December 2016
the whole month →
Talking face-to-face with your team is just one click away
Work · 2016 · slack.com
What's your workplace flavor? Join 10M+ users and take the free personality test to find out. Discover your workplace strengths and ideal career path in under 3 minutes with our CPTI™ assessment, based on the Myers-Briggs framework.
Dev tools · 2016 · cake.me
