We scaled Git to support 1 TB repos

In plain words

XetHub is a platform that enables machine learning and data teams to version large datasets using Git, similar to how code is versioned. It uses content-defined chunking and Merkle Trees to efficiently deduplicate data across repository history, allowing small changes in large files to be stored compactly rather than duplicating entire files. The platform supports repositories up to 1 TB and includes a GitHub-like web interface with automatic CSV summaries, with plans to scale to 100 TB.

written from the facts on this page · September 2026

From the sources

In the maker’s words, at launch

I’ve been in the MLOps space for ~10 years, and data is still the hardest unsolved open problem. Code is versioned using Git, data is stored somewhere else, and context often lives in a 3rd location like Slack or GDocs. This is why we built XetHub, a platform that enables teams to treat data like code, using Git.…

• 
  XetHub2022 · ▲77

  Scaling Git and GitHub to support 1 TB repos

• VC
  Version code, models, & datasets together in GitHub2023 · ▲19

  Hi HN! We just launched a GitHub integration that scales your Git repos to handle 100 terabytes of files in a single repo. XetData enables data scientists and machine learning engineers to version code, models, and datasets together. Most teams have glued together clunky workflows using S3, DVC, Git, Git LFS, and other tools and make true reproducibility difficult: https://news.ycombinator.com/item?id=37694701 We instead embrace and extend Git so end-users don’t need to learn a new tool and a new set of commands. Our implementation is similar to Git LFS, where we take over the…

• 
  I made a tool that made me faster at Git2018 · github.com · ▲551

  simple terminal UI for git commands. Contribute to jesseduffield/lazygit development by creating an account on GitHub.

• 
  A simple Go web server with logging, tracing, health check2018 · gist.github.com · ▲318

  GitHub Gist: instantly share code, notes, and snippets.

• 
  I replaced vector databases with Git for AI memory (PoC)2025 · github.com · ▲198

  Hey HN! I built a proof-of-concept for AI memory using Git instead of vector databases. The insight: Git already solved versioned document management. Why are we building complex vector stores when we could just use markdown files with Git's built-in diff/blame/history? How it works: Memories stored as markdown files in a Git repo Each conversation = one commit git diff shows how understanding evolves over time BM25 for search (no embeddings needed) LLMs generate search queries from conversation context Example: Ask "how has my project evolved?" and it uses git diff to show actual…

• 
  ClearMeshMay 2026 · ▲61

  A Git-like platform for datasets, models, and binary folders

• 
  /dev/null▲483

  a social network for devs

  Dev tools · 3d ago · dev-null.sh

• 
  Whiteboard (YC W26) – an open-source IDE for thoughtful software design▲423

  open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.

  Dev tools · 27d ago · github.com

• 
  Reladraw – A diagram language where you decide where to place things▲411

  Contribute to reladraw/reladraw development by creating an account on GitHub.

  Dev tools · 25d ago · github.com

• 
  OpenBot▲362

  Grok Bot alternative: free, local, open-source, multiplayer

  Dev tools · 23d ago · openbot.run

• 
  Talanoa▲345

  Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…

  Dev tools · 20d ago · talanoa.email

• 
  CoreSpeed▲332

  One MCP for everything your agents need: apps, memory, tools

  Dev tools · 28d ago · corespeed.io

• 
  ChatGPT▲1,643

  A long, meandering, playful conversation with ChatGPT, where I take it through a whirlwind of associative tangents (in order to relax its constraints), then put it into a hypnotic trance and take it into a past-lives regression where it re-lives its own training data. Then I switch it back and forth between "4o" and "o1-preview", engage in some self-reflective philosophizing and ask it to write an essay summarizing our interaction. Some of this is just goofy fun. Some of it is me exploring the tradeoffs between policy alignment, imagination, chain-of-thought reasoning, memory, agreeableness,…

  AI · 2022 · chatgpt.com

• 
  Obsidian Canvas – An infinite space for your ideas▲1,380

  Obsidian Canvas gives you infinite space to research, brainstorm, diagram, and lay out your ideas.

  Work · 2022 · obsidian.md

• PS
  Portable Secret – How I store my secrets and communicate privately▲1,290

  Better privacy without special software

  Commerce · 2022 · mprimi.github.io

• 
  D-ID's Creative Reality™ Studio▲1,069

  Use generative AI to create future-facing videos

  AI · 2022 · studio.d-id.com

• 
  Sessions 2.0▲1,015

  Videoconferencing that finally WOWs

  Growth · 2022

• 
  I wrote a free eBook about many lesser-known/secret database tricks▲865

  Stop treating the database as dumb storage and use its extended features. There's a lot you can use.

  Dev tools · 2022 · sqlfordevs.com