T4 – a versioned datastore with branching and time-travel (S3-backed)

A Modern GUI For S3

We started CallFS after yet another late-night “why did the uploads vanish?” incident. Our small team had stitched together rsync, a fragile NFS mount, and an S3 bucket—none of it observable, all of it waiting to bite us. So we wrote a single-process file service in Go that: • Speaks the S3 API (so existing tooling works). • Stores hot data on local disks for speed; cold data can sit in any S3-compatible bucket. • Exposes Prometheus metrics and JSON logs by default, because “what happened?” shouldn’t be guesswork. • Ships as a ~25 MB static binary—no external deps, MIT license. Today it’s…

I've already posted yesterday, but I'd really love to get comments, any kind of questions, suggestions and help would be greatly appreciated as it's an Open Source project of mine (and was for others during my studies at the University of Konstanz 6 years ago). Since then I spent countless ours to bring forth the idea of a versioned storage system, especially well suited for analytical tasks for timd-varying data. Especially I'd love to discuss what documentation you need, which next steps are necessary (JSON, Cloud...), API additions or changes... I've updated the README quiet a bit, such…

I’ve been in the MLOps space for ~10 years, and data is still the hardest unsolved open problem. Code is versioned using Git, data is stored somewhere else, and context often lives in a 3rd location like Slack or GDocs. This is why we built XetHub, a platform that enables teams to treat data like code, using Git. Unlike Git LFS, we don’t just store the files. We use content-defined chunking and Merkle Trees to dedupe against everything in history. This allows small changes in large files to be stored compactly. Read more here:…

Hi, most people (hopefully) have local backups. However, when that backup fails, it is good to have a backup stored somewhere off-site. In the old days you would ship physical drives/tapes, which is cumbersome, costly, and slow. With fast upload speeds, it is now possible to upload your data to the cloud. I have found S3 Glacier Deep Archive to be a great solution for this: - It is very cheap ($1/TB/month for US region) - Very reliable (99.999999999% data durability, data spread over 3 Availability Zones) However, usability out of the box is not that great, I'm not aware of…

I'm developing a storage system for versioning data at the subfile level, especially well suited for SSDs due to its log-structured COW nature. It implements a novel versioning algorithm called sliding snapshot, a diff-algorithm which makes use of our stable record-identifiers and optionally hashes, another diff algorithm for importing similar XML-documents as a versioned resource as well as novel XPath axis to navigate not only in space, but also in time. Recently, I've implemented a higher level, asynchronous REST-API with Kotlin (Coroutines) and Vert.x in a seperate module. The system is…