Alternatives
Products that do what DeltaGlider – Store 4TB of build artifacts in 5GB does
DeltaGlider is a CLI/SDK similar to `aws s3` or `boto3`. UPLOAD: It stores the first file in a S3 path as a full-size (reference), but saves next uploaded archives as deltas (tiny binary diffs) with respect to the reference. DOWNLOAD: it reconstructs the original file on the fly, bit-perfect and verified with SHA256. Why Xdelta3? It's a compression-aware and block-level binary diff algorithm. Perfect for representing differences between archives, where small changes shift bytes but most content stays the same. It can efficiently delta compress ZIP/JAR/TAR archives up to 99.9%…
- 1CS
We started CallFS after yet another late-night “why did the uploads vanish?” incident. Our small team had stitched together rsync, a fragile NFS mount, and an S3 bucket—none of it observable, all of it waiting to bite us. So we wrote a single-process file service in Go that: • Speaks the S3 API (so existing tooling works). • Stores hot data on local disks for speed; cold data can sit in any S3-compatible bucket. • Exposes Prometheus metrics and JSON logs by default, because “what happened?” shouldn’t be guesswork. • Ships as a ~25 MB static binary—no external deps, MIT license. Today it’s…
2025 · github.com
- 2WS
I’ve been in the MLOps space for ~10 years, and data is still the hardest unsolved open problem. Code is versioned using Git, data is stored somewhere else, and context often lives in a 3rd location like Slack or GDocs. This is why we built XetHub, a platform that enables teams to treat data like code, using Git. Unlike Git LFS, we don’t just store the files. We use content-defined chunking and Merkle Trees to dedupe against everything in history. This allows small changes in large files to be stored compactly. Read more here:…
2022 · xethub.com
- 3
File-based memory for OpenClaw with >92% retrieval accuracy
Mar 2026
- 4
- 5TA
I built a SQLite VFS in Rust that serves cold queries directly from S3 with sub-second performance, and often much faster. It’s called turbolite. It is experimental, buggy, and may corrupt data. I would not trust it with anything important yet. I wanted to explore whether object storage has gotten fast enough to support embedded databases over cloud storage. Filesystems reward tiny random reads and in-place mutation. S3 rewards fewer requests, bigger transfers, immutable objects, and aggressively parallel operations where bandwidth is often the real constraint. This was explicitly inspired…
Mar 2026 · github.com
- 6ZA
Jul 2026 · zerofs.net
- 7SL
I've created a powerful tool to help Go developers uncover the hidden giants in their compiled binaries. Go Size Analyzer is like an X-ray machine for your Go executables, revealing: Which dependencies are eating up your binary size Unexpected bloat from standard library or vendor packages Size changes between binary versions with a visual diff Key features that set it apart: Interactive treemap visualizations (check out the demo: https://gsa.zxilly.dev) Slick terminal UI for deep diving into package hierarchies Cross-platform support (works on Linux, macOS, and Windows binaries)…
2024 · github.com
- 8NS
Hello! This is Edgar and Robbie and we built nix-snapshotter. nix-snapshotter brings native understanding of Nix packages to containerd. We built this because Nix is a great fit for making efficient containers. They don't need an OS because Nix captures all dependencies exactly. However, the current process of creating Nix images is subpar because one needs to transform Nix packages into a format that container runtimes understand. Using nix-snapshotter, instead of downloading image layers, packages come directly from the Nix store. Packages can be fetched from a binary cache or built on the…
2023 · github.com
- 9DB
My partner reviews a lot of P&IDs (piping and instrumentation diagrams) and the adjacent files involved (excel, docx, pdfs, acd/l5x, etc). In his company, these are usually done in Bluebeam. It's really hard to see the diff + keep track of all the revisions resulted by these iterations. They end up storing files like "rev3_final_redlined.pdf". We've been looking for something close to Github to do these reviews, but haven't found one easy enough for folks with no CLI experience to understand and use (happy to check out more tools if you know any). So I built withkord.com to help with…
Aug 2026 · withkord.com
- 10
- 11
- 12YA
Built this for my LLM workflows - needed searchable, persistent memory that wouldn't blow up storage costs. I also wanted to use it locally for my research. It's a content-addressed storage system with block-level deduplication (saves 30-40% on typical codebases). I have integrated the CLI tool into most of my workflows in Zed, Claude Code, and Cursor, and I provide the prompt I'm currently using in the repo. The project is in C++ and the build system is rough around the edges but is tested on macOS and Ubuntu 24.04.
2025 · github.com
- 13HE
I wrote a little LD_PRELOAD library that makes it easy to inspect and interact with a running program's glibc heap. It's fun to pause processes, free a bunch of their allocations, then resume them. Most of the time, the processes continue as though nothing happened, but sometimes they do interesting things :)
2025 · github.com
- 14OA
I built an experiment that uses an overfitted transformer and arithmetic coding to compress individual files. Instead of training the model to generalize, I train a 900KB transformer to memorize a single file and predict the next byte. Those predictions are fed into an arithmetic coder to produce the compressed output. On a 100MB NYC taxi CSV, it compresses to about 7MB (~0.5 bits/byte). On a 100MB slice of enwik9, it compresses to about 21MB (~1.68 bits/byte). It's pretty slow right now (roughly 20–30 minutes of training and 45 minutes each for compression and decompression on my…
Jun 2026
- 15AS
Hey everyone, I’m Hunter, the founder of Archil. Archil is transforming object storage, like Amazon S3, into infinite, local file systems that provide instant access to massive data sets. Last year, we launched Archil’s NFS-based product publicly on Hacker News (https://news.ycombinator.com/item?id=42174204), and we were absolutely thrilled to see the response of this community. Since our last launch, we took a 10 month company-wide bet to build our own, custom storage protocol to deliver true, local-like performance to cloud instances – by behaving closer to a block storage…
Sep 2025
- 16
This project is a little performance-optimization experiment focused on reducing the size of Go binary patches, which can be useful for quick incremental updates of live services. When a Go program grows by a few bytes through an incremental change, a bunch of related references and offsets throughout the compiled binary end up changing that makes general-purpose binary-diff encoders inefficient: `zstd --patch-from` produces a 530kb patch for a one-line change in a 30MB binary. bsdiff improves upon this (150KB patch) thanks to a delta compression algorithm that can handle relative offsets.…
10d ago · github.com
- 17SA
SlateDB is an embedded storage engine built as a log-structured merge-tree. Unlike traditional LSM-tree storage engines, SlateDB writes data to object storage (S3, GCS, ABS, MinIO, Tigris, and so on). Leveraging object storage allows SlateDB to provide bottomless storage capacity, high durability, and easy replication. The trade-off is that object storage has a higher latency and higher API cost than local disk. To mitigate high write API costs (PUTs), SlateDB batches writes. Rather than writing every put() call to object storage, MemTables are flushed periodically to object storage as a…
2024 · github.com
- 18ZT
Sep 2025 · github.com
- 19BD
2020 · github.com
- 20BH
Usually, I’m a researcher working on areas like RL for coding agents – but one day I became annoyed by our slow VM startup times. So I took the plunge into systems engineering and built the first version of our VM hypervisor called otterlink. It now powers both our research & all of Devin production workloads. Compared to EC2, otterlink was able to bring down VM startup times by about 10x. The real pain point, however, were EC2’s long snapshot times. We want a lot of flexibility (e.g. forking, rollback and suspending VMs) that all require taking disk snapshots. On EC2 taking disk snapshots…
2025 · cognition.ai
- 21SV
I've already posted yesterday, but I'd really love to get comments, any kind of questions, suggestions and help would be greatly appreciated as it's an Open Source project of mine (and was for others during my studies at the University of Konstanz 6 years ago). Since then I spent countless ours to bring forth the idea of a versioned storage system, especially well suited for analytical tasks for timd-varying data. Especially I'd love to discuss what documentation you need, which next steps are necessary (JSON, Cloud...), API additions or changes... I've updated the README quiet a bit, such…
2018
- 22SS
I'm developing a storage system for versioning data at the subfile level, especially well suited for SSDs due to its log-structured COW nature. It implements a novel versioning algorithm called sliding snapshot, a diff-algorithm which makes use of our stable record-identifiers and optionally hashes, another diff algorithm for importing similar XML-documents as a versioned resource as well as novel XPath axis to navigate not only in space, but also in time. Recently, I've implemented a higher level, asynchronous REST-API with Kotlin (Coroutines) and Vert.x in a seperate module. The system is…
2018
- 23IC
This is an upgrade of a tool I created 15 years ago in Python to learn OOP and solve some inadequacies in the HDR stacking tools I could find at the time. The problem was, none of them were really "batch friendly". None of them properly preserved the metadata I wanted them to stuff into the output file. There were probably some other reasons also, I just can't remember them now. It got the job done, but was very slow. Python was what I knew at the time and even with NumPy, I was limited in the speed I could squeeze out of it. (I also made some very specific, conscious, architectural choices…
Jun 2026 · github.com
- 24BB
This project addresses the inefficiencies of traditional Dockerfile-based container builds where each customization layer creates storage bloat through duplicate dependencies from repeated apt-get install commands, network inefficiency from redundant package downloads across different images, and slow iteration cycles requiring full rebuilds of all previous steps. Our solution enables building minimal base images from scratch using debootstrap that precisely include only required components in the initial build, while allowing creation of specialized variants (Java, Kafka, etc.) from these…
2025 · github.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →