We scaled Git to support 1 TB repos

What it does

In the maker’s words, at launch

I’ve been in the MLOps space for ~10 years, and data is still the hardest unsolved open problem. Code is versioned using Git, data is stored somewhere else, and context often lives in a 3rd location like Slack or GDocs. This is why we built XetHub, a platform that enables teams to treat data like code, using Git. Unlike Git LFS, we don’t just store the files. We use content-defined chunking and Merkle Trees to dedupe against everything in history. This allows small changes in large files to be stored compactly. Read more here: https://xethub.com/assets/docs/how-xet-deduplication-works Today, XetHub works for 1 TB repositories, and we plan to scale to 100 TB in the next year. Our implementation is in Rust (client & cache + storage) and our web application is written in Go. XetHub includes a GitHub-like web interface that provides automatic CSV summaries and allows custom visualizations using Vega. Even at 1 TB, we know downloading an entire repository is painful, so we built git-xet mount - which, in seconds, provides a user-mode filesystem view over the repo. XetHub is available today (Linux & Mac today, Windows coming soon) and we would love your feedback! Read more here: - https://xetdata.com/blog/2022/10/15/why-xetdata - https://xetdata.com/blog/2022/12/13/introducing-xethub

• AV
  A version control system based on rsync2023 · jamsync.dev · ▲249

  Hi everyone! I'm trying to create a version control system that solves some of the problems that Git and other version control software has when working in a team. Let me know if you have any feedback!

• GT
  Gitorials – Tutorials as Git Repos2017 · gitorials.com · ▲147
• VC
  Version code, models, & datasets together in GitHub2023 · ▲19

  Hi HN! We just launched a GitHub integration that scales your Git repos to handle 100 terabytes of files in a single repo. XetData enables data scientists and machine learning engineers to version code, models, and datasets together. Most teams have glued together clunky workflows using S3, DVC, Git, Git LFS, and other tools and make true reproducibility difficult: https://news.ycombinator.com/item?id=37694701 We instead embrace and extend Git so end-users don’t need to learn a new tool and a new set of commands. Our implementation is similar to Git LFS, where we take over the…

• HF
  Hubfs – File System for GitHub2022 · github.com · ▲160
• GA
  GitPlex – A new Git repo management server with code review2016 · pmease.com · ▲193
• AV
  A versioned filesystem inspired by Git2012 · github.com · ▲115

• 
  Dograh▲592

  The open source VAPI alternative

  Dev tools · 25d ago · dograh.com

• 
  Meridian▲530

  Don't let your work go unnoticed. Get promoted!

  Dev tools · 20d ago · meridiona.com

• 
  x1▲516

  Lovable for iPhone apps go from idea to App Store

  Dev tools · 11d ago · x1.new

• 
  The GTM Co-Founder▲402

  Open-source GTM skills for technical founders

  Dev tools · 29d ago · gtmcofounder.com

• 
  Nuphos▲380

  The AI-Native DevOps Workspace.

  Dev tools · 24d ago · nuphos.ai

• 
  Lettertrace▲375

  Track your AI visibility for free (using your own API keys!)

  Dev tools · 25d ago · lettertrace.com

• 
  ChatGPT▲1,647

  Optimizing language models for dialogue

  AI · 2022 · chatgpt.com

• OC
  Obsidian Canvas – An infinite space for your ideas▲1,380

  Life & fun · 2022 · obsidian.md

• PS
  Portable Secret – How I store my secrets and communicate privately▲1,290

  Commerce · 2022 · mprimi.github.io

• 
  D-ID's Creative Reality™ Studio▲1,072

  Use generative AI to create future-facing videos

  AI · 2022 · studio.d-id.com

• 
  Sessions 2.0▲1,015

  Videoconferencing that finally WOWs

  Growth · 2022

• IW
  I wrote a free eBook about many lesser-known/secret database tricks▲865

  Dev tools · 2022 · sqlfordevs.com