I used streaming to skip downloading my 45GB dataset
In plain words
This GitHub project demonstrates a technique for accessing large datasets without downloading them entirely to local storage. By using streaming methods, users can work with a 45GB dataset while minimizing disk space requirements. The tool is useful for researchers, data scientists, and developers who need to process large files but have limited storage capacity or slow download speeds. It was released in November 2022.
written from the facts on this page · September 2026
Does a similar job
all alternatives →- SGStreaming gigabyte medical images from S3 without downloading themJan 2026 · github.com · ▲161
- WSWe scaled Git to support 1 TB repos2022 · xethub.com · ▲279
I’ve been in the MLOps space for ~10 years, and data is still the hardest unsolved open problem. Code is versioned using Git, data is stored somewhere else, and context often lives in a 3rd location like Slack or GDocs. This is why we built XetHub, a platform that enables teams to treat data like code, using Git. Unlike Git LFS, we don’t just store the files. We use content-defined chunking and Merkle Trees to dedupe against everything in history. This allows small changes in large files to be stored compactly. Read more here:…
- SSStreambed – Stream Postgres to Iceberg on S3, Supports Postgres WireMay 2026 · github.com · ▲129
- OAOverfitted a 900KB Transformer to Compress a 100MB CSV into 7MBJun 2026 · ▲112
I built an experiment that uses an overfitted transformer and arithmetic coding to compress individual files. Instead of training the model to generalize, I train a 900KB transformer to memorize a single file and predict the next byte. Those predictions are fed into an arithmetic coder to produce the compressed output. On a 100MB NYC taxi CSV, it compresses to about 7MB (~0.5 bits/byte). On a 100MB slice of enwik9, it compresses to about 21MB (~1.68 bits/byte). It's pretty slow right now (roughly 20–30 minutes of training and 45 minutes each for compression and decompression on my…
More life & fun this month
the category →- TL
Life & fun · 11d ago · louisabraham.github.io
Photosynthesis fires two of your iPhone
Life & fun · 29d ago · photosynthesis.camera
- CCCreatium Coach▲320
Your multimedia mentor that takes you from mid to great
Life & fun · 11d ago · producthunt.creatium.info
SoloUno▲310Take control of hair pulling, nail biting & skin picking
Life & fun · 29d ago · solouno.io
Launched alongside, November 2022
the whole month →
- TC
I made a thing! In 2014, I was holding a stack of iPhones and thought to myself: "Hey, if I had each phone display a playing card, I could click a button and they'd shuffle themselves" I pared that idea all the way down to this: trading cards made of e-ink displays. Right now, each card costs me about $20 each, but with only a bit more scale, I think I can get that down to $10. In doing this project, I learned how to design electronics and circuit boards. I learned Rust and wrote my first driver, I upped my CAD skills, 3D printed, and did my first resin casting. I generated the images on the…
Dev tools · 2022 · wyldcard.io
