jj-benchmark – Evaluating AI agents on Jujutsu version control
Hi HN, Meng from TabbyML here. We decided to build this simply because we find Jujutsu (jj) really interesting, and many folks on our team have started trying it out recently. Since it introduces a very different workflow compared to traditional Git, we thought it would be a fun challenge to see how well current AI coding agents can actually use it. To build this, we created a semi-automated pipeline. We used AI to research the official Jujutsu documentation and websites, which then helped us bootstrap a dataset of 63 distinct evaluation tasks. Each task includes instructions, bootstrap…
What it does
In the maker’s words, at launch
Hi HN, Meng from TabbyML here. We decided to build this simply because we find Jujutsu (jj) really interesting, and many folks on our team have started trying it out recently. Since it introduces a very different workflow compared to traditional Git, we thought it would be a fun challenge to see how well current AI coding agents can actually use it. To build this, we created a semi-automated pipeline. We used AI to research the official Jujutsu documentation and websites, which then helped us bootstrap a dataset of 63 distinct evaluation tasks. Each task includes instructions, bootstrap scripts, and tests. We then ran the evaluations using the Harbor framework and our Pochi agent. Some interesting insights from our initial leaderboard: Claude 4.6 Sonnet is the clear winner: It achieved a 92% success rate (passing 58/63 tasks), beating out Opus and OpenAI's top models. It seems exceptionally good at parsing the novel CLI rules of jj. The Speed vs. Accuracy Trade-off: While GPT-5.4 sits at #5 with an 81% success rate, it is incredibly fast, averaging just 77.6s per task. In contrast, Gemini-3.1-pro achieved 84% but took over 3x as long (267.6s average). Open Weights / Regional Models are competitive: Models like Kimi-k2.5 (79%) put up a very respectable fight on a relatively niche tool. The benchmark isn't completely solved yet, but the fact that top models can successfully navigate a relatively new version control system by reasoning through the tasks is pretty exciting. If there are specific jj edge cases you think we should add to the dataset, feel free to open up a PR!
Does the same job
all alternatives →
- LALoadjitsu – a modern load testing alternative to JMeter2022 · loadjitsu.com · ▲67
Please meet Loadjitsu, my weekend project, years in the making. Over the years while building different apps and sites, I always felt that I need a modern load testing software.Tools like JMeter, ab are not very easy to use and it seems innovation in load testing which is a crucial part of any software release cycle has been ignored. This is my third attempt at making Loadjitsu, I am so glad that I can finaly release this. A bit more about the software 1. Powered by golang you can run load tests for tens of thousands of connections per second on very average hardware. 2. Cross platform, run…
- ARA registry of agent benchmarks (including many OSS agent trajectories)2024 · explorer.invariantlabs.ai · ▲6
If you're interested in exploring what LLM-based agent systems these days actually do to solve certain benchmarks such as SWEBench or WebArena, we created a small leaderboard with our team, that allows to view a lot of public and OSS agent results including all the runtime traces (the step-by-step reasoning behind the scenes). Looking at traces is actually quite interesting, as they reveal a lot about the inner working and shortcomings of current agent system, e.g. see https://explorer.invariantlabs.ai/u/invariant/webarena--SteP... for an example trace.
- LOLegit, Open source Git-based Version control for AI agentsJan 2026 · ▲9
Hi HN, Martin, Nils, and Jannes here. We are building Legit, an open source version control and collaboration layer for AI agents and AI native applications. You can find the repo here https://github.com/Legit-Control/monorepo and the website here https://legitcontrol.com Over the last years, we worked on multiple developer tools and AI driven products. As soon as we started letting agents modify real files and business critical data, one problem kept showing up. We could not reliably answer what changed, why it changed, or how to safely undo it. Today, most AI…
- SRShikigami, run AI coding agents in parallel, each in a Git worktreeJul 2026 · shikigami.dev · ▲7
Hello all, I'm a software developer. Over the last few months more and more of my work has turned into using coding agents instead of typing the whole code myself. Usually a few claude sessions at once, sometimes codex, one per feature or per revealed bug. I ran them in a split terminal for a few weeks, and quickly spotted two main problems. The first is that I couldn't easily tell which agent was stuck waiting on me and which was still working, so I'd cycle through sessions and checking on them. The second one: agents sharing a single branch step on each other. Two of them could be editing…
- MDML-Dev-Bench – Benchmarking AI Agents on Real-World AI Workflows2025 · github.com · ▲6
We’re excited to share ML-Dev-Bench, a new open-source benchmark that tests AI agents on real-world ML development tasks. Unlike typical coding challenges or Kaggle-style competitions, our benchmark simulates end-to-end ML workflows including: - Dataset handling and preprocessing - Debugging model and code failures - Implementing new model architectures - Fine-tuning and improving existing models With 30 diverse tasks, ML-Dev-Bench evaluates agents across critical stages of ML development. To complement this, we built Calipers, a framework that provides systematic performance evaluation and…
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 26d ago · cactuscompute.com
Launched alongside, March 2026
the whole month →
Switch from ChatGPT to Claude with import memory feature
AI · Mar 2026 · claude.com
