Dev tools · alternatives · 2026
24 alternatives to OnPair – String compression with fast random access (Rust, C++)
I’ve been working on a compression algorithm for fast random access to individual strings in large collections. The problem came up when working with large in-memory database columns (emails, URLs, product titles, etc.), where low-latency point queries are essential. With short strings, LZ77-based compressors don’t perform well. Block compression helps, but block size forces a trade-off between ratio and access speed. Some existing options: - BPE: good ratios, but slow and memory-heavy - FSST (discussed here: https://news.ycombinator.com/item?id=41489047): very fast, but…
Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes.
- 1
Hey there HN! We’re Antonio, Luca, and Yuhang, and we’re excited to introduce Fast GraphRAG, an open-source RAG approach that leverages knowledge graphs and the 25 years old PageRank for better information retrieval and reasoning. Building a good RAG pipeline these days takes a lot of manual optimizations. Most engineers intuitively start from naive RAG: throw everything in a vector database and hope that semantic search is powerful enough. This can work for use cases where accuracy isn’t too important and hallucinations are tolerable, but it doesn’t work for more difficult queries that…
2024 · github.com · its alternatives →
- 2AF
Serialization framework with some interesting numbers: 10-20x faster on nested objects than json/protobuf. Technical approach: compile-time codegen (no reflection), compact binary protocol with meta-packing, little-endian layout optimized for modern CPUs. Unique features that other fast serializers don't have: - Cross-language without IDL files (Rust ↔ Python/Java/Go) - Trait object serialization (Box) - Automatic circular reference handling - Schema evolution without coordination Happy to discuss design trade-offs. Benchmarks:…
Oct 2025 · fory.apache.org · its alternatives →
- 3
A vector index built on TurboQuant, written in Rust with Python bindings - RyanCodrai/turbovec
Mar 2026 · github.com · its alternatives →
- 4OA
I built an experiment that uses an overfitted transformer and arithmetic coding to compress individual files. Instead of training the model to generalize, I train a 900KB transformer to memorize a single file and predict the next byte. Those predictions are fed into an arithmetic coder to produce the compressed output. On a 100MB NYC taxi CSV, it compresses to about 7MB (~0.5 bits/byte). On a 100MB slice of enwik9, it compresses to about 21MB (~1.68 bits/byte). It's pretty slow right now (roughly 20–30 minutes of training and 45 minutes each for compression and decompression on my…
Jun 2026 · its alternatives →
- 5
Tantivy▲68A full-text, horse-speed search engine library in Rust
2022 · github.com · its alternatives →
- 6
LoopMix128 is a fast C PRNG I wrote for non-cryptographic tasks. GitHub (MIT): https://github.com/danielcota/LoopMix128 Highlights: * ~0.37 ns/value (GCC 11.4, -O3 -march=native), 98% faster than xoroshiro128++ and PCG64. * Passes TestU01 BigCrush & PractRand (32TB). * Guaranteed 2^128 period. * Proven injective (192-bit state) via Z3 SMT solver; allows parallel streams. * Core requires only stdint.h. Seeking feedback on design, use cases, or further testing.
2025 · github.com · its alternatives →
- 7
HNSW implementation in Rust. Reference: https://arxiv.org/ftp/arxiv/papers/1603/1603.09320.pdf - swapneel/hnsw-rust
2024 · github.com · its alternatives →
- 8
I built a SQLite VFS in Rust that serves cold queries directly from S3 with sub-second performance, and often much faster. It’s called turbolite. It is experimental, buggy, and may corrupt data. I would not trust it with anything important yet. I wanted to explore whether object storage has gotten fast enough to support embedded databases over cloud storage. Filesystems reward tiny random reads and in-place mutation. S3 rewards fewer requests, bigger transfers, immutable objects, and aggressively parallel operations where bandwidth is often the real constraint. This was explicitly inspired…
Mar 2026 · github.com · its alternatives →
- 9
I want to share a really dumb, but very practical project I have packaged this summer, to perform operations on strings much faster. I was using Python to work with a multi-terabyte newline-delimited file. Reading, splitting, and shuffling it was a nightmare. So, I wrapped a trivial hardware-friendly heuristic I've been using for the last few years into a CPython library. The part I enjoyed the most is implementing SIMD behavior without SIMD instructions... Using 64-bit words to work at 8-bit granularity. Unlike conventional SIMD, the code would remain the same for ~~almost~~ any hardware.…
2023 · ashvardanian.com · its alternatives →
- 10
tl;dr we built an embeddable stream processing engine in Rust using apache DataFusion, check us out at https://github.com/probably-nothing-labs/denormalized Hey HN, We’d like to showcase a very early version of our embeddable stream processing engine called Denormalized. The rise of DuckDB has abundantly made it clear that even for many workloads of Terabyte scale, a single node system outshines the distributed query engines of previous generation such as Spark, Snowflake etc in terms of both performance and cost. Now a lot of workloads DuckDB is used for were normally…
2024 · github.com · its alternatives →
- 11
A fast and memory-optimized string library for C++ - btwael/SuperString
2018 · github.com · its alternatives →
- 12
Rapid fuzzy string matching in Python using various string metrics - rapidfuzz/RapidFuzz
2020 · github.com · its alternatives →
- 13
Hi HN! I've spent the last few weeks optimizing the string operations in StringZilla. You'd be surprised how much performance GLibc and the Standard Templates Library of C++ are leaving on the table, especially on Arm and, likely, on RISC-V. I've significantly increased the number of implemented algorithms for different workloads. Searching across English texts on Arm, for example: - is 4.4x faster than STL, - and 16.8x faster in reverse order. The new version of the library now includes a C++ binding largely compatible with the C++ Standard, featuring all 17 overloads of the `replace`…
2024 · github.com · its alternatives →
- 14
While building a Retrieval-Augmented Generation (RAG) system, I was frustrated by my vector database consuming 8GB RAM just to search my own PDFs. After incurring $150 in cloud costs, I had an unconventional idea: what if I encoded my documents into video frames? The concept sounded absurd—storing text in video? But modern video codecs have been optimized for compression over decades. So, I converted text into QR codes, then encoded those as video frames, letting H.264/H.265 handle the compression. The results were surprising. 10,000 PDFs compressed down to a 1.4GB video file. Search…
2025 · github.com · its alternatives →
- 15CD
2017 · excamera.com · its alternatives →
- 16
Hi HN, I’m building SEE (Semantic Entropy Encoding): a searchable compression format for JSON/NDJSON. Goal: reduce the “data tax” (storage/egress) and “CPU tax” (decompress/parse) by keeping JSON searchable while compressed, with page-level random access. I just published a proof-first evaluation release: Offline DEMO ZIP (~10 min): prints compression ratios + skip rates + lookup latency (p50/p95/p99) DD pack: audit/repro evidence (decode mismatch=0, extended mismatch=0, audit PASS) Latest release:…
Feb 2026 · gitlab.com · its alternatives →
- 17
Hello! I'm learning Rust and also information security, so what better way to learn than to put a tool together practicing both skills? I've tried to use a couple of different tools for subdomain enumeration (part of the Discovery-party when it comes to penetration testing), but none of them were very fast, even if the DNS resolver I pointed them towards is, so I wrote my own tool that leverages concurrency to check a lot of subdomains, fast! It's written in Rust, main parts being using Tokio and async_channels, and I think it works good enough for a first release. I wanted to get recursive…
2022 · github.com · its alternatives →
- 18
I've been working on Fast LiteLLM - a Rust acceleration layer for the popular LiteLLM library - and I had some interesting learnings that might resonate with other developers trying to squeeze performance out of existing systems. My assumption was that LiteLLM, being a Python library, would have plenty of low-hanging fruit for optimization. I set out to create a Rust layer using PyO3 to accelerate the performance-critical parts: token counting, routing, rate limiting, and connection pooling. The Approach - Built Rust implementations for token counting using tiktoken-rs - Added lock-free data…
Nov 2025 · github.com · its alternatives →
- 19
A very fast scalar implementation for Frame Of Reference integer compression. - cruppstahl/libfor
2015 · github.com · its alternatives →
- 20
Hey HN, My friend Sam and I were bored one weekend, and wanted to make something self-contained but still useful. We ended up with Victor [0], a vector database designed to run in the browser. It’s written in Rust, which gives us access to things like super-fast and compact serialization and deserialization, but it’s compiled to WASM and uses web-sys to interact with the browser’s JS APIs to write the database contents to disk. We worked hard on designing it in a way that makes sense for the browser. It uses the private origin file system [1] and stores the vectors in a compact format (one…
2023 · github.com · its alternatives →
- 21
2018 · dropbox.github.io · its alternatives →
- 22AS
Took a stab at building my own string diffing JS package. I built an interactive demo for TypeIt’s website (https://typeitjs.com/build-your-own) a while back. The approach I took to handle user input necessitated a way to calculate the difference between the versions of a user’s text input. I searched around for a package to help me out and found a couple of good ones (like fast-diff), but I either didn’t really like their API or didn’t want to take on a huge new dependency. Instead, I thought I’d give it a shot myself (famous last words). I dove into it having no real formal…
2022 · its alternatives →
- 23
Happy to release FastPlaid, which aim to ease and accelerate ColBERT and ColPali retrieval
2025 · github.com · its alternatives →
- 24
Scalable, Low-latency and Hybrid-enabled Vector Search in Postgres. Revolutionize Vector Search, not Database. - tensorchord/pgvecto.rs
2023 · github.com · its alternatives →
Also compare
- FastGraphRAG – Better RAG using good old PageRank alternatives
- Apache Fory Rust – 10-20x faster serialization than JSON/Protobuf alternatives
- TurboQuant for vector search – 2-4 bit compression alternatives
- Overfitted a 900KB Transformer to Compress a 100MB CSV into 7MB alternatives
- Tantivy alternatives
- LoopMix128 – Fast C PRNG (.46ns), 2^128 Period, BigCrush/PractRand Pass alternatives
Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →