All-pair similarity search on millions of sets in Python and on laptop

In plain words

This is a Python developer tool for finding similar sets among millions of items, designed to run on standard laptops. It performs all-pair similarity searches efficiently without requiring specialized hardware or cloud infrastructure. The tool is useful for developers working with large datasets who need to identify duplicate or comparable sets quickly. It was released in 2018 and is available on GitHub.

written from the facts on this page · September 2026

• 
  How to analyse 100 GB of data on your laptop with Python2019 · towardsdatascience.com · ▲264

  Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.

• 
  Fast vector similarity using Rust and Python2023 · github.com · ▲141

  I recently found myself computing the similarity between lots of very high dimensional vectors (i.e., sentence embedding vectors from LLMs), and I wanted to try some more powerful measures of similarity/dependency than just Cosine similarity, which seems to be the default for everything nowadays because of its computational efficiency. There are many other more involved measures that can detect more subtle relationships, but the problem is that some of them are quite slow to compute, especially if you're trying to do it in Python. For my favorite measure of statistical dependency,…

• 
  Scalable reverse image search built on Kubernetes and Elasticsearch2016 · github.com · ▲139

  :crystal_ball: Scalable reverse image search built on Kubernetes and Elasticsearch - dsys/match

• 
  On-disk B+ tree for Python 32018 · github.com · ▲159

  An on-disk B+tree for Python 3. Contribute to NicolasLM/bplustree development by creating an account on GitHub.

• 
  RapidFuzz – A fast string matching library for Python2020 · github.com · ▲134

  Rapid fuzzy string matching in Python using various string metrics - rapidfuzz/RapidFuzz

• DD
  Data Diff – compare tables of any size across databases2022 · ▲127

  Gleb, Alex, Erez and Simon here – we are building an open-source tool for comparing data within and across databases at any scale. The repo is at https://github.com/datafold/data-diff, and our home page is https://datafold.com/. As a company, Datafold builds tools for data engineers to automate the most tedious and error-prone tasks falling through the cracks of the modern data stack, such as data testing and lineage. We launched two years ago with a tool for regression-testing changes to ETL code…

• 
  Run full iOS 16.5 firmware on Android Devices▲508

  Contribute to oosexgang/hackios-xnu-kernel development by creating an account on GitHub.

  Dev tools · 1d ago · github.com

• 
  /dev/null▲492

  a social network for devs

  Dev tools · 4d ago · dev-null.sh

• 
  Whiteboard (YC W26) – an open-source IDE for thoughtful software design▲423

  open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.

  Dev tools · 29d ago · github.com

• 
  Reladraw – A diagram language where you decide where to place things▲411

  Contribute to reladraw/reladraw development by creating an account on GitHub.

  Dev tools · 27d ago · github.com

• 
  OpenBot▲362

  Grok Bot alternative: free, local, open-source, multiplayer

  Dev tools · 24d ago · openbot.run

• 
  Talanoa▲346

  Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…

  Dev tools · 22d ago · talanoa.email

• 
  Krisp Mac▲893

  Record outgoing calls and get transcripts + AI notes

  AI · 2018 · krisp.ai

• 
  Web.dev by Google▲815

  Learn how to build for the web and see where you stand 🌟

  Dev tools · 2018 · web.dev

• 
  DrawKit▲765

  MIT licensed SVG illustrations, in 2 styles

  Dev tools · 2018

• 
  The 1.1.1.1 App▲689

  Faster, more private internet by Cloudflare, now on mobile

  Work · 2018 · apps.apple.com

• 
  TabNine, an autocompleter for all languages▲607

  Tabnine is the AI code assistant that accelerates and simplifies software development while keeping your code private, secure, and compliant.

  AI · 2018 · tabnine.com

• 
  1Kprojects▲585

  Neglected side projects for less than $1000.

  Dev tools · 2018 · indiemaker.com