Version code, models, & datasets together in GitHub
Hi HN! We just launched a GitHub integration that scales your Git repos to handle 100 terabytes of files in a single repo. XetData enables data scientists and machine learning engineers to version code, models, and datasets together. Most teams have glued together clunky workflows using S3, DVC, Git, Git LFS, and other tools and make true reproducibility difficult: https://news.ycombinator.com/item?id=37694701 We instead embrace and extend Git so end-users don’t need to learn a new tool and a new set of commands. Our implementation is similar to Git LFS, where we take over the…
In plain words
XetData is a GitHub integration that allows data scientists and machine learning engineers to version control code, models, and datasets together in a single repository, supporting up to 100 terabytes of files. It extends Git's functionality similarly to Git LFS, eliminating the need to learn additional tools or combine multiple services like S3 and DVC. The platform includes automated pull request comments with dataset views and model comparisons to enhance collaboration and reproducibility.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
Hi HN! We just launched a GitHub integration that scales your Git repos to handle 100 terabytes of files in a single repo. XetData enables data scientists and machine learning engineers to version code, models, and datasets together. Most teams have glued together clunky workflows using S3, DVC, Git, Git LFS, and other tools and make true reproducibility difficult: https://news.ycombinator.com/item?id=37694701 We instead embrace and extend Git so end-users don’t need to learn a new tool and a new set of commands. Our implementation is similar to Git LFS, where we take over the .gitattributes file, push pointers to large files in GitHub, and push the raw, large files to us. We have a few distinct features that we’re proud of that improve the user experience: - Our XetData bot comments on your pull requests to provide links to useful dataset views and model diffs. We’re working on rendering these inside GitHub itself using browser extensions. - Git LFS and similar tools only implement file-level deduplication. We created a new technique called block-based deduplication (published in CIDR’23 conference) specifically for data and ML workflows. The ML lifecycle consists of making lots of iterative changes and our technique helps save storage and time spent downloading and uploading changes. - You can mount large repos to your local machine using git-xet mount for exploratory work. Individual files that are needed are streamed in just in time behind the scenes. We open sourced our implementation of mount and it was well received here on HN: https://news.ycombinator.com/item?id=37573679 - To give more users access to your data, just add them to your GitHub repo. This is a beta product and we would love all of your feedback. You can find all instructions to try this out here: https://github.com/apps/xetdata While we’re in beta, our product is completely free to use. We have a Slack you can join or a GitHub issue tracker. - Slack: https://communityinviter.com/apps/xetdata/xet - GitHub: https://github.com/xetdata/xet-tools/issues/
Does the same job
all alternatives →- WSWe scaled Git to support 1 TB repos2022 · xethub.com · ▲279
I’ve been in the MLOps space for ~10 years, and data is still the hardest unsolved open problem. Code is versioned using Git, data is stored somewhere else, and context often lives in a 3rd location like Slack or GDocs. This is why we built XetHub, a platform that enables teams to treat data like code, using Git. Unlike Git LFS, we don’t just store the files. We use content-defined chunking and Merkle Trees to dedupe against everything in history. This allows small changes in large files to be stored compactly. Read more here:…
- IRI replaced vector databases with Git for AI memory (PoC)2025 · github.com · ▲198
Hey HN! I built a proof-of-concept for AI memory using Git instead of vector databases. The insight: Git already solved versioned document management. Why are we building complex vector stores when we could just use markdown files with Git's built-in diff/blame/history? How it works: Memories stored as markdown files in a Git repo Each conversation = one commit git diff shows how understanding evolves over time BM25 for search (no embeddings needed) LLMs generate search queries from conversation context Example: Ask "how has my project evolved?" and it uses git diff to show actual…
- IVInstantly visualize any codebase as an interactive diagram2024 · gitdiagram.com · ▲222
GitDiagram is an open-source micro dev-tool that I made this past week Given any public GitHub repository it generates diagrams in Mermaid.js with Claude 3.5 Sonnet I extract information from the file tree and README for details and interactivity (you can click components to be taken to relevant files and directories) Also, you can replace "hub" with "diagram" in any repository URL to access its diagram I created this because I wanted to contribute to open-source projects but quickly realized their codebases are too massive for me to dig through manually, so this helps me get started I do…
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 27d ago · cactuscompute.com
Launched alongside, November 2023
the whole month →
Discover & book top creators to promote your product
Growth · 2023 · passionfroot.me
