I used streaming to skip downloading my 45GB dataset

In plain words

This GitHub project demonstrates a technique for accessing large datasets without downloading them entirely to local storage. By using streaming methods, users can work with a 45GB dataset while minimizing disk space requirements. The tool is useful for researchers, data scientists, and developers who need to process large files but have limited storage capacity or slow download speeds. It was released in November 2022.

written from the facts on this page · September 2026

• 
  Streaming gigabyte medical images from S3 without downloading themJan 2026 · github.com · ▲161

  A tile server for WSI stored in S3-compatible object storage. - PABannier/WSIStreamer

• 
  Dagster+2024 · dagster.io · ▲286

  Ship data pipelines with extraordinary velocity

• 
  How to analyse 100 GB of data on your laptop with Python2019 · towardsdatascience.com · ▲264

  Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.

• 
  Estuary Flow2023 · estuary.dev · ▲225

  Connect your data, where you want it, in milliseconds

• WS
  We scaled Git to support 1 TB repos2022 · xethub.com · ▲279

  I’ve been in the MLOps space for ~10 years, and data is still the hardest unsolved open problem. Code is versioned using Git, data is stored somewhere else, and context often lives in a 3rd location like Slack or GDocs. This is why we built XetHub, a platform that enables teams to treat data like code, using Git. Unlike Git LFS, we don’t just store the files. We use content-defined chunking and Merkle Trees to dedupe against everything in history. This allows small changes in large files to be stored compactly. Read more here:…

• FZ
  Free, zero configuration hosting for data and models using Git and DVC2021 · ▲7

  Hey HN! I'm Guy, one of the creators of DAGsHub. We help data scientists track experiments and version their code, data & models using Git. We're committed to building on open source protocols and improving the user experience on top of them - in this case, DVC. We've built capabilities on top of it like visually interacting with your pipeline, creating data pull requests, browsing data files etc. One thing we kept running into is users being challenged by setting up the cloud environment necessary, to push their data to AWS S3, GCS (which we support), and other DVC remotes. Since our focus…

• 
  Run full iOS 16.5 firmware on Android Devices▲514

  Contribute to oosexgang/hackios-xnu-kernel development by creating an account on GitHub.

  Dev tools · 1d ago · github.com

• 
  /dev/null▲492

  a social network for devs

  Dev tools · 4d ago · dev-null.sh

• 
  Whiteboard (YC W26) – an open-source IDE for thoughtful software design▲423

  open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.

  Dev tools · 29d ago · github.com

• 
  Reladraw – A diagram language where you decide where to place things▲411

  Contribute to reladraw/reladraw development by creating an account on GitHub.

  Dev tools · 27d ago · github.com

• 
  OpenBot▲362

  Grok Bot alternative: free, local, open-source, multiplayer

  Dev tools · 24d ago · openbot.run

• 
  Talanoa▲346

  Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…

  Dev tools · 22d ago · talanoa.email

• 
  VEED.IO▲1,623

  Simple online video editing

  Growth · 2022 · veed.io

• 
  Notion AI (Alpha)▲1,483

  Meet your new partner in thought

  AI · 2022 · app.notion.com

• 
  Rewind▲1,420

  The search engine for your life

  AI · 2022 · rewind.ai

• 
  Trading cards made with e-ink displays▲1,149

  I made a thing! In 2014, I was holding a stack of iPhones and thought to myself: "Hey, if I had each phone display a playing card, I could click a button and they'd shuffle themselves" I pared that idea all the way down to this: trading cards made of e-ink displays. Right now, each card costs me about $20 each, but with only a bit more scale, I think I can get that down to $10. In doing this project, I learned how to design electronics and circuit boards. I learned Rust and wrote my first driver, I upped my CAD skills, 3D printed, and did my first resin casting. I generated the images on the…

  Dev tools · 2022 · wyldcard.io

• 
  Taplio▲1,004

  The all-in-one tool to grow your personal brand on LinkedIn

  Growth · 2022 · taplio.com

• 
  BeforeSunset▲915

  Catch beautiful sights, finish your work before sunset

  Work · 2022