Alternatives
Products that do what I compressed 10k PDFs into a 1.4GB video for LLM memory does
While building a Retrieval-Augmented Generation (RAG) system, I was frustrated by my vector database consuming 8GB RAM just to search my own PDFs. After incurring $150 in cloud costs, I had an unconventional idea: what if I encoded my documents into video frames? The concept sounded absurd—storing text in video? But modern video codecs have been optimized for compression over decades. So, I converted text into QR codes, then encoded those as video frames, letting H.264/H.265 handle the compression. The results were surprising. 10,000 PDFs compressed down to a 1.4GB video file. Search…
- 1OA
I built an experiment that uses an overfitted transformer and arithmetic coding to compress individual files. Instead of training the model to generalize, I train a 900KB transformer to memorize a single file and predict the next byte. Those predictions are fed into an arithmetic coder to produce the compressed output. On a 100MB NYC taxi CSV, it compresses to about 7MB (~0.5 bits/byte). On a 100MB slice of enwik9, it compresses to about 21MB (~1.68 bits/byte). It's pretty slow right now (roughly 20–30 minutes of training and 45 minutes each for compression and decompression on my…
Jun 2026
- 2MO
Hey HN, we’re Adi and Arnav. A few months ago, we hit a wall trying to get LLMs to answer questions over research papers and instruction manuals. Everything worked fine, until the answer lived inside an image or diagram embedded in the PDF. Even GPT‑4o flubbed it (we recently tried O3 with the same, and surprisingly it flubbed it too). Naive RAG pipelines just pulled in some text chunks and ignored the rest. We took an invention disclosure PDF (https://drive.google.com/file/d/1ySzQgbNZkC5dPLtE3pnnVL2rW_9...) containing an IRR‑vs‑frequency graph and asked GPT “From…
2025 · github.com
- 3
- 4BC
I know there are various tools that are supposed to make this easy, but I couldn't find anything that did everything I wanted, so I made this today for fun. The web-based offerings all take forever and seem flaky, and you need to process one video at a time, with no control over the transcription settings. In contrast, my script lets you convert a whole playlist in bulk with full control over everything. It's truly easy to use-- you can clone the repo, install to a venv, and be generating a folder full of high quality transcript text files in under 5 minutes. All you need to do is supply the…
2023 · github.com
- 5
- 6
- 76E
Hi HN, I’ve been experimenting with archive format design and built 6cy as a research project. The goal is not to replace zip/7z, but to explore: • block-level codec polymorphism (different compression per block) • streaming-first layout (no global seek required) • better crash recovery characteristics • plugin-based architecture so proprietary codecs can exist without changing the format Right now this is an experimental v0.x format. The specification may still change and compatibility is not guaranteed yet. I’m mainly looking for feedback on the format design rather than performance…
Feb 2026 · github.com
- 8
- 9CA
Thought I'd dive into this cool streaming/approximation algorithms problem I encountered a few months ago. TLDR: we can create our own custom floating point representation for encoding and decoding integers and use this to index into a tiny 2D histogram, upper-bounding approximation error based on the number of bits per integer we specify to keep. --- Broadly useful for aggregating statistics from massive data streams of user data. Also turns out this is incredibly similar to data structures used in production, like HDRHistogram.
Jan 2026 · alexkranias.com
- 10IC
This is an upgrade of a tool I created 15 years ago in Python to learn OOP and solve some inadequacies in the HDR stacking tools I could find at the time. The problem was, none of them were really "batch friendly". None of them properly preserved the metadata I wanted them to stuff into the output file. There were probably some other reasons also, I just can't remember them now. It got the job done, but was very slow. Python was what I knew at the time and even with NumPy, I was limited in the speed I could squeeze out of it. (I also made some very specific, conscious, architectural choices…
Jun 2026 · github.com
- 11
- 12RA
RAGLite is a Python package for building Retrieval-Augmented Generation (RAG) applications. RAG applications can be magical when they work well, but anyone who has built one knows how much the output quality depends on the quality of retrieval and augmentation. With RAGLite, we set out to unhobble RAG by mapping out all of its subproblems and implementing the best solutions to those subproblems. For example, RAGLite solves the chunking problem by partitioning documents in provably optimal level 4 semantic chunks. Another unique contribution is its optimal closed-form linear query adapter…
2024 · github.com
- 13LA
Hi HN! Vectara is a "batteries included" retrieval augmented generation platform. You can upload your rich text documents like PDFs, HTML pages, word docs, etc, or semi-structured JSON and Vectara handles the text and metadata extraction, segmentation, vector embedding, and vector storage, and keyword storage. You can ask a question or perform a search in the UI or via our APIs and Vectara will automatically handle the vectorization, structured metadata filtering, vector+keyword retrieval, hybrid blending, and generative summarization of the results. We're focusing on building and…
2023 · vectara.com
- 14SM
2021 · acailly.github.io
- 15QS
Of course this is just for fun
2024 · github.com
- 16OV
2024 · github.com
- 17LA
Hey everyone, I’ve been working on a project called L88 — a local RAG system that I initially focused on UI/UX for, so the retrieval and model architecture still need proper refinement. Repo: https://github.com/Hundred-Trillion/L88-Full I’m running this on 8GB VRAM and a strong CPU (128GB RAM). Embeddings and preprocessing run on CPU, and the main model runs on GPU. One limitation I ran into is that my evaluator and generator LLM ended up being the same model due to compute constraints, which defeats the purpose of evaluation. I’d really appreciate feedback on:…
Feb 2026
- 18
- 19IC
9d ago · figshare.com
- 20BA
Hey HN, solo dev here. After years of frustration with how LLMs handle complex documents, especially PDFs with tables, I decided to build a solution myself. My approach uses a Markdown conversion step to preserve the table structure, which seems to work surprisingly well for chunking. This little parser is the first public piece of a much larger, privacy-focused AI platform I'm building. I'm pretty much running on fumes financially, so any feedback, critique, or support is massively appreciated. Happy to answer any questions about the approach!
Nov 2025 · github.com
- 21
- 22IS
Read an article yesterday about the H.264 codec increasing their licensing fee by an astronomical amount. And as always, my first shot was how hard could it be to try and build a codec which could be that efficient. I've personally been on a drive to improve my ability to one-shot complex features, products, or make even surgical changes. It's been a few months since I've been doing that, and honestly, results have been great for both work and work/life balance. This was a fun experiment. It burned through tokens, but it helped me identify some more improvements I could make to my…
Apr 2026 · github.com
- 23SC
Problem JSON/NDJSON is everywhere in data platforms, but compression usually breaks searchability. You either keep queryable raw stores (high I/O/egress) or compress into gz/zstd blobs (cheap to store, painful to probe). The “cloud tax” shows up as wasted reads. What I built (SEE — Semantic Entropy Encoding) A schema-aware, searchable compression codec for JSON that keeps exists/pos lookups fast while still compressing. Internals: structure-aware delta + dictionaries, a PageDir + mini-index to jump to relevant pages, and a tuned Bloom filter that skips ~99% of pages.…
Oct 2025 · github.com
- 24
Ranked by how close each launch is in meaning, then by votes. Refine with a description →