Alternatives
Products that do what Compressing a Histogram from 16GB to 2KB does
Thought I'd dive into this cool streaming/approximation algorithms problem I encountered a few months ago. TLDR: we can create our own custom floating point representation for encoding and decoding integers and use this to index into a tiny 2D histogram, upper-bounding approximation error based on the number of bits per integer we specify to keep. --- Broadly useful for aggregating statistics from massive data streams of user data. Also turns out this is incredibly similar to data structures used in production, like HDRHistogram.
- 1FI
2015 · github.com
- 2IM
2019 · github.com
- 3
- 4AI
2017 · unwttng.com
- 5
- 6H2
2016 · github.com
- 7BD
2020 · github.com
- 8TB
2015 · github.com
- 9SL
I'd like to share a little toy project of mine, a really simple image codec that can do lossy to full lossless image compression with complete scalability at a byte level granularity - you can compress an image just once, even fully losslessly if needed, and then get any lossy version possible by simply stopping decompression at any offset in the compressed data. This "encode onde, serve many" approach is especially interesting for providing downscaled low quality image previews (LQIP) in as tight a storage budget as possible, and then allowing seamless, transparent refinement as deemed…
2024 · github.com
- 10CI
Author: Danison Nuñez Project: Coderive Project Type: Programming Language Highlight: Formula-based Execution ## Coderive - Iterating Through 1 Quintillion in a Loop *Subtitle: How a phone interpreter achieves what supercomputers cannot* The Impossible Loop: ```java // In any other language, this would be computational suicide for i in [0 to 1Qi] { // 1,000,000,000,000,000,000 iterations arr[i] = i * i } ``` Traditional Reality: · Python: MemoryError at array creation · Java/C++: Theoretical 31 years (with 8 exabytes of RAM) · NumPy/TensorFlow: Immediate crash ·…
Dec 2025 · github.com
- 11AN
Kimi K3 has 2.78 trillion parameters and ships as 1.42 TB of weights. It clearly does not fit in the memory of a laptop. But K3 is a Mixture-of-Experts model. For each token, only a small fraction of its 896 experts per layer is activated. That changes the problem: the entire model does not need to be resident in RAM, as long as the weights required by each token can be reached quickly enough. We built WASTE — the Weight-Aware Streaming Tensor Engine — to explore that idea. WASTE keeps the dense, repeatedly used part of the model resident in memory, stores the routed experts in an…
Jul 2026
- 12HA
Most ZK proving systems are optimized for server-grade hardware with massive RAM. When scaling to industrial-sized traces (2^20+ rows), they often hit a "Memory Wall" where allocation and data movement become a larger bottleneck than the actual computation. I have been developing Hekate, a ZK engine written in Rust that utilizes a Zero-Copy streaming model and a hybrid tiled evaluator. To test its limits, I ran a head-to-head benchmark against Binius64 on an Apple M3 Max laptop using Keccak-256. The results highlight a significant architectural divergence: At 2^15 rows: Binius64 is faster…
Jan 2026
- 13AA
2014 · tapcelerate.com
- 14SS
Hi HN, I’m building SEE (Semantic Entropy Encoding): a searchable compression format for JSON/NDJSON. Goal: reduce the “data tax” (storage/egress) and “CPU tax” (decompress/parse) by keeping JSON searchable while compressed, with page-level random access. I just published a proof-first evaluation release: Offline DEMO ZIP (~10 min): prints compression ratios + skip rates + lookup latency (p50/p95/p99) DD pack: audit/repro evidence (decode mismatch=0, extended mismatch=0, audit PASS) Latest release:…
Feb 2026 · gitlab.com
- 15DA
Hi HN, I'd like to share DualMix128, a pseudo-random number generator I developed. It's written in C and achieves very high speed while maintaining strong statistical properties for non-cryptographic use. GitHub (MIT License): https://github.com/the-othernet/DualMix128 Key points: * *Fast:* Benchmarked at ~0.36 ns per 64-bit generation on GCC 11.4 (-O3 -march=native). This was over 2x faster than `xoroshiro128++` (~0.74 ns) and competitive with `wyrand` (~0.36 ns) in the same tests. (Benchmark code/results in repo). * *Robust:* Passes the full TestU01 BigCrush suite…
2025 · github.com
- 16MM
Hi HN! We (Thomas and Stéphan, hello!) recently released Model2Vec, a Python library for distilling any sentence transformer into a small set of static embeddings. This makes inference with such a model up to 500x faster, and reduces model size by a factor of 15 (7.5M params or 15/30MB on disk, depending on whether you use float16 or float32). This allows you to embed 50-100k documents per second on a cpu on a macbook. This reduction of course comes at a cost: distilled models are worse than their parent models. Even so, they are actually a lot better than large sets of conventional…
2024 · github.com
- 17IM
Jan 2026 · github.com
- 18OS
I’ve been working on a compression algorithm for fast random access to individual strings in large collections. The problem came up when working with large in-memory database columns (emails, URLs, product titles, etc.), where low-latency point queries are essential. With short strings, LZ77-based compressors don’t perform well. Block compression helps, but block size forces a trade-off between ratio and access speed. Some existing options: - BPE: good ratios, but slow and memory-heavy - FSST (discussed here: https://news.ycombinator.com/item?id=41489047): very fast, but…
2025 · github.com
- 19HA
2018 · github.com
- 20MA
I've been working on training this small vision language model for the last month - excited to release the first prototype today! It is based on SigLIP (image encoder), Phi-1.5 (text model) and trained using the LLaVa-1.5 training dataset. It runs reasonably fast on CPU with ~8GB of RAM in full 32-bit precision. There's plenty of room to speed it up and reduce memory consumption by quantizing the model. I posted a video of it running on my M2 Macbook Air (on CPU not MPS, so performance should be comparable on other hardware) on Twitter to demonstrate inference speed:…
2023 · github.com
- 21AB
hey there hn! thought you would appreciate my brand new file format - i've spent over 40 minutes creating this standard. it's really hard trying to get your ideas shown to standards boards etc but i think with the great community here we can help get bioif support into all major browsers :) the spec is available over at the website, as is a reference implementation. it already has higher usage than jpeg-xl. - best
2025 · image.bsb.dev
- 22T1
2020 · github.com
- 23AV
AvifHash leverages the power of AVIF to create image placeholders that are both compact and efficient. This Proof of Concept shows promising results: at 27 characters, AvifHash outperforms BlurHash https://blurha.sh/ (using 4x3 components) in quality and detail retention. At a similar quality, BlurHash needs 54 (5x5) to 76 characters (6x6 components). Given that AVIF decoding is done by the web engine, AvifHash is very small: the entire demo page (including parsing and re-hydration code) is only 2.3 kB gzipped.
2024 · juliobbv.com
- 24
I made this after seeing someone posit the idea online yesterday over lunch then spent some time refining it. So far it's pretty impressive IMO! Right now I am running Qwen3-30B-A3B on my 24gb unified memory m4 MacBook Pro at 50 tok/sec and this should definitely not be working for such a large model on my middling hardware. Things are detailed in the README to get up and running and DESIGN.md has details on all the choices and such made along the way.
23d ago · github.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →