Dev tools · alternatives · 2026
24 alternatives to I used streaming to skip downloading my 45GB dataset
DagsHub client libraries. Contribute to DagsHub/client development by creating an account on GitHub.
This GitHub project demonstrates a technique for accessing large datasets without downloading them entirely to local storage. By using streaming methods, users can work with a 45GB dataset while minimizing disk space… Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes. I used streaming to skip downloading my 45GB dataset launched in 2022; newer entries below may have overtaken it.
- 1
A tile server for WSI stored in S3-compatible object storage. - PABannier/WSIStreamer
Jan 2026 · github.com · its alternatives →
- 2
- 3
Your home for data science and AI. The world's leading publication for data science, data analytics, data engineering, machine learning, and artificial intelligence professionals.
2019 · towardsdatascience.com · its alternatives →
- 4
Estuary Flow▲225Connect your data, where you want it, in milliseconds
2023 · estuary.dev · its alternatives →
- 5WS
I’ve been in the MLOps space for ~10 years, and data is still the hardest unsolved open problem. Code is versioned using Git, data is stored somewhere else, and context often lives in a 3rd location like Slack or GDocs. This is why we built XetHub, a platform that enables teams to treat data like code, using Git. Unlike Git LFS, we don’t just store the files. We use content-defined chunking and Merkle Trees to dedupe against everything in history. This allows small changes in large files to be stored compactly. Read more here:…
2022 · xethub.com · its alternatives →
- 6FZ
Hey HN! I'm Guy, one of the creators of DAGsHub. We help data scientists track experiments and version their code, data & models using Git. We're committed to building on open source protocols and improving the user experience on top of them - in this case, DVC. We've built capabilities on top of it like visually interacting with your pipeline, creating data pull requests, browsing data files etc. One thing we kept running into is users being challenged by setting up the cloud environment necessary, to push their data to AWS S3, GCS (which we support), and other DVC remotes. Since our focus…
2021 · its alternatives →
- 7
Stream Postgres to Apache Iceberg on S3 via logical replication, queryable over the Postgres wire protocol. - GitHub - viggy28/streambed: Stream Postgres to Apache Iceberg on S3 via logical replication, queryable over the Postgres wire protocol.
May 2026 · github.com · its alternatives →
- 8
A totally proof-of-concept FoundationDB based network block device backend - dividuum/fdb-nbd
2018 · github.com · its alternatives →
- 9
Publish and share huge 3D datasets online
2018 · sketchfab.com · its alternatives →
- 10
- 11OA
I built an experiment that uses an overfitted transformer and arithmetic coding to compress individual files. Instead of training the model to generalize, I train a 900KB transformer to memorize a single file and predict the next byte. Those predictions are fed into an arithmetic coder to produce the compressed output. On a 100MB NYC taxi CSV, it compresses to about 7MB (~0.5 bits/byte). On a 100MB slice of enwik9, it compresses to about 21MB (~1.68 bits/byte). It's pretty slow right now (roughly 20–30 minutes of training and 45 minutes each for compression and decompression on my…
Jun 2026 · its alternatives →
- 12
Over the weekend I was discussing the latest news relating to open weights with some of my non-techie friends. After 30 mins of waffling, I realised it is quite a long and interesting story, with many twists and turns. I've always wanted to create a 'long scroll' story, but have never found a suitable story to tell. This felt like a fun weekend project to undertake. The story itself doesn't have too many scroll effects (which is probably a good thing), it was more about trying to tell a story with a certain style that wouldn't really fit on my personal blog. Would love to get your feedback…
2020 · blog.scottlogic.com · its alternatives →
- 13
Hey there! This is Dan and Ustin (@uzarubin), and we want to share something cool we've been working on for the past year - an open-source `tail -f` for your data, with a UI. We call it "Streamdal" which is a word salad for streaming systems (because we love them) and DAL or data access layer (because we’re nerds). Here's the repo: https://github.com/streamdal/streamdal Here's the site: https://streamdal.com And here's a live demo: https://demo.streamdal.com (github repo has an explanation of the demo) — — — THE PROBLEM We built this because the…
2023 · github.com · its alternatives →
- 14
Optimal Quantile Approximation in Streams. Contribute to axiomhq/quantiles development by creating an account on GitHub.
2018 · github.com · its alternatives →
- 15
Cloudflare R2▲127Zero data transfer fees object storage
2022 · developers.cloudflare.com · its alternatives →
- 16
Hi HN! We are Ashish and Armend, founders of GlassFlow. We just launched our open-source streaming ETL that deduplicates and joins Kafka streams before ingesting them to ClickHouse https://github.com/glassflow/clickhouse-etl Why we built this: Dedup with batch data is straightforward. You load the data into a temporary table. Then, find only the latest versions of the record through hashes or keys and keep them. After that, move the clean data into your main table. But have you tried this with streaming data? Users of our prev product were running real-time analytics…
2025 · github.com · its alternatives →
- 17
tl;dr we built an embeddable stream processing engine in Rust using apache DataFusion, check us out at https://github.com/probably-nothing-labs/denormalized Hey HN, We’d like to showcase a very early version of our embeddable stream processing engine called Denormalized. The rise of DuckDB has abundantly made it clear that even for many workloads of Terabyte scale, a single node system outshines the distributed query engines of previous generation such as Spark, Snowflake etc in terms of both performance and cost. Now a lot of workloads DuckDB is used for were normally…
2024 · github.com · its alternatives →
- 18
Hey HN! I just released go-nbd, a lightweight Go library for effortlessly creating NBD servers and clients. Its a neat tool for creating custom Linux block devices with arbitrary backends, such as a file, byte slice or what I'm planning to use it for, a tape drive. While there are a few partially abandoned projects like this out there already, this library tries to be as maintainable as possible by only implementing the most recent handshake revision and baseline functionality for both the client and the server, while still having enough support to be useful. I'd love to get your feedback :)
2023 · github.com · its alternatives →
- 19
BStreams▲62Interactive, easy, responsive data visualization & analysis
2021 · bstreams.io · its alternatives →
- 20
serverless cross-platform streaming and aggregation
2022 · github.com · its alternatives →
- 21
- 221S
2011 · mortardata.com · its alternatives →
- 23
Hi HN. Today we released Dagster Cloud to general availability [1], which includes a new feature you can try that we're calling Branch Deployments. Branch Deployments were inspired by Vercel's Preview Deployments feature and deep GitHub integration. We're hoping we can bring a similar developer experience improvement to the data domain. Let us know your feedback! [1] https://dagster.io/blog/dagster-cloud-ga-launch
2022 · twitter.com · its alternatives →
- 24
Experiments that involve encoding arbitrary data into video files that survive Youtube compression. - ianling/steg-experiments
2025 · github.com · its alternatives →
Also compare
- Streaming gigabyte medical images from S3 without downloading them alternatives
- Dagster+ alternatives
- How to analyse 100 GB of data on your laptop with Python alternatives
- Estuary Flow alternatives
- We scaled Git to support 1 TB repos alternatives
- Free, zero configuration hosting for data and models using Git and DVC alternatives
Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →