nowfound

AI · June 22, 2020

D1

DVC 1.0 release. 5 lessons from 3 years of building open-source ML tool

Hey HN, creator of DVC here! DVC (https://dvc.org/) is known as Git for data projects. Technically, DVC codifies your data and machine learning pipelines as text metafiles (with pointers to actual data in S3/GCP/Azure/SSH) while you use Git for the actual versioning. DevOps folks call this approach GitOps or more specifically in this case - DataOps or MLOps. We’ve been working towards 1.0 since we started 3 years ago. What began as my pet project now has 100+ code contributors, 100+ documentation contributors, and thousands of users. Our community has taught us…

Alternatives

What it does

In the maker’s words, at launch

Hey HN, creator of DVC here! DVC (https://dvc.org/) is known as Git for data projects. Technically, DVC codifies your data and machine learning pipelines as text metafiles (with pointers to actual data in S3/GCP/Azure/SSH) while you use Git for the actual versioning. DevOps folks call this approach GitOps or more specifically in this case - DataOps or MLOps. We’ve been working towards 1.0 since we started 3 years ago. What began as my pet project now has 100+ code contributors, 100+ documentation contributors, and thousands of users. Our community has taught us a lot - here are some of the biggest lessons: 1. Users say the serverless and distributed nature of DVC (inherited from the underlying Git) is one of its "killer features". 2. To share ML projects within and between teams, it’s not enough to track only files and pipelines. You also need metrics, plot and hyperparameter tracking. In DVC 1.0 we implemented hyper-parameter diffs, metrics and plot diffs right from Git history. 3. In DataOps, data transfer optimization is huge. Large deep learning models, millions of images in datasets, etc. We doubled down on optimizing 1.0. 4. ML pipelines evolve faster than data engineering pipelines and need to be easy to change. In 1.0, we’ve simplified the pipeline metafile format. 5. More and more teams use DVC as a part of CI/CD for ML and other MLOps tools. DVC is used under the hood in the CD4ML tool that was described in the canonical post on Martin Fowler’s blog: https://martinfowler.com/articles/cd4ml.html. We built 1.0 with CI/CD users in mind. More details on https://dvc.org/blog/dvc-1-0-release. Happy to answer any questions here or at DVC Discord chat https://dvc.org/chat.

Does the same job

all alternatives →
  • DVC Studio2021 · ▲95

    Machine Learning Experiments based on Git

  • DVC Extension for VS Code2023 · ▲165

    Track machine learning experiments right in your IDE

  • FZ
    Free, zero configuration hosting for data and models using Git and DVC2021 · ▲7

    Hey HN! I'm Guy, one of the creators of DAGsHub. We help data scientists track experiments and version their code, data & models using Git. We're committed to building on open source protocols and improving the user experience on top of them - in this case, DVC. We've built capabilities on top of it like visually interacting with your pipeline, creating data pull requests, browsing data files etc. One thing we kept running into is users being challenged by setting up the cloud environment necessary, to push their data to AWS S3, GCS (which we support), and other DVC remotes. Since our focus…

  • Git analytics2019 · ▲131

    for software developers, teams, and open-source communities

  • Datature2021 · ▲271

    No-code platform for building deep neural nets

  • vitMar 2026 · ▲186

    git for video editing.

More ai this month

the category →
  • I trained a 125M model to autocomplete piano on-device▲598

    I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.

    AI · 17d ago · simedw.com

  • Astute▲585

    Automate your B2B brand going viral, with new media creators

    AI · 18d ago · company-app.joinastute.com

  • Grok Bot▲547

    AI teammates that you can give real work to

    AI · 25d ago · x.ai

  • Needle2: 14MB agentic LLM for phones, wearables, smart home and robots▲537

    Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…

    AI · 27d ago · cactuscompute.com

  • Clara AI SDR▲460

    Turn website visitors into qualified pipeline

    AI · 19d ago · clarasdr.ai

  • Kane CLI▲446

    Natural language browser & mobile app tests from terminal

    AI · 24d ago · testmuai.com

Launched alongside, June 2020

the whole month →