OpenBenchmarks – Helping agents discover and pick the right SaaS APIs
I'm Fenil, co-founder/CEO of OpenFunnel (YC F24), building this with my co-founder/CTO Aditya. We're launching OpenBenchmarks (https://openbenchmarks.com), open-source, reproducible benchmarks for SaaS APIs, starting with the category we know best: GTM APIs. ## Why we built this More and more B2B software evaluation will/already runs through reasoning models inside agentic workflows rather than through people. And buyers increasingly pick vendors that are API-first and ship MCPs, so they can wire them into internal workflows. Strong reasoning models are skeptical of…
What it does
In the maker’s words, at launch
I'm Fenil, co-founder/CEO of OpenFunnel (YC F24), building this with my co-founder/CTO Aditya. We're launching OpenBenchmarks (https://openbenchmarks.com), open-source, reproducible benchmarks for SaaS APIs, starting with the category we know best: GTM APIs. ## Why we built this More and more B2B software evaluation will/already runs through reasoning models inside agentic workflows rather than through people. And buyers increasingly pick vendors that are API-first and ship MCPs, so they can wire them into internal workflows. Strong reasoning models are skeptical of marketing. When a genuinely neutral benchmark is available, they discount SEO and self-published benchmarks, and they lose trust the moment something looks like a marketing claim. The thing that survives that skepticism is an independent build-first benchmark the agent can reproduce itself and trust. Reproducible by default: every cell ships the literal HTTP request/response plus the judge prompt/response All benchmarks live at https://github.com/openbenchmarks-labs. We start with GTM APIs, with more on the way. ## On Benching ourselves We benched our own product (OpenFunnel) as a vendor in the lookalikes benchmark, on purpose. We increasingly saw buyers asking for benchmarks on sales calls and were also curious to see if agents also made decisions the same way. We wanted to see whether an agent could run the whole loop end to end: discover the benchmark while researching a user's query, weigh it as the deciding factor in picking a winner for that user's category, then sign up and auth through to the chosen vendor to complete the task. That last stretch needs live vendors with agent-auth wired up: us and a few others. We came out #1 on the current seed (89% vs 74%). Because it's open, we don't win everything: depending on seed and metric (precision@10/@50/@100), we win some and lose others. ## Dogfooding We dogfooded it and ran 200 incognito simulated buyer flows through Claude Code with live web search, from query to discovery to sign-up, and watched which sources the model fetched and which ones made it into the final decision. The benchmark got opened in a large majority of runs, across the range of queries an actual buyer would ask in Claude Code, from building a lookalikes workflow to picking a lookalike API provider. And when it was opened, it usually drove the final call, over self-published GEO pages and vendor benchmarks with years of domain authority. We've studied the effect and potential ROI of being benched, we're taking OpenFunnel off the benchmark. ## What's next More GTM benchmarks, then beyond: devtools and infrastructure. We're are also working with traditional SaaS companies thinking about going API-first and opening up to a new class of customer: agents Try it, or hand it to your agent: https://openbenchmarks.com
Does the same job
all alternatives →
- OBOpen Benchmarks Grants– a $3M commitment to close the AI eval gapFeb 2026 · benchmarks.snorkel.ai · ▲6
Today, we're launching the Open Benchmarks Grants: a $3M commitment to fund open-source and academic teams building benchmarks for AI agents. In partnership with HuggingFace, PrimeIntellect, FactoryHQ, Together, Harbor, and PyTorch, the grants provide funding, data development support, and research collaboration. Our ability to measure AI has been outpaced by our ability to develop it, and we believe this evaluation gap is one of the most important problems in AI. Open benchmarks are one of the most important levers for advancing AI safely and responsibly—but the academic and open-source…
- ARA registry of agent benchmarks (including many OSS agent trajectories)2024 · explorer.invariantlabs.ai · ▲6
If you're interested in exploring what LLM-based agent systems these days actually do to solve certain benchmarks such as SWEBench or WebArena, we created a small leaderboard with our team, that allows to view a lot of public and OSS agent results including all the runtime traces (the step-by-step reasoning behind the scenes). Looking at traces is actually quite interesting, as they reveal a lot about the inner working and shortcomings of current agent system, e.g. see https://explorer.invariantlabs.ai/u/invariant/webarena--SteP... for an example trace.
- TRTrunchbull, run real models against any benchmark in your browser25d ago · trunchbull.dev · ▲6
Hi HN, Today I'm showcasing Trunchbull, a benchmarking platform designed for authoring benchmarks and running them against different models. We have direct support for benchmarks that use the harbor authoring system, custom tool authoring via the vercel ai sdk and configuration limits. We've also already imported terminalbench 2.0, as a sort of proof of concept that our harbor task orchestrator works, although you currently need a paid account as we are provisioning sandbox environments. I've made several popular benchmarks publicly available for testing. You dont need an account or your…
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 1d ago · opentrailpaper.com
Launched alongside, July 2026
the whole month →- IR
I might be the only SRE on Earth with his own bowling center. It's a more in-depth gig than you'd think. My family and I bought an abandoned 8-lane bowling center in the rural mid-west. In our small town there weren't many recreation options for families. You've heard of a food desert? This is an R&R desert. It had been abandoned for a good reason. The roof leaks, the electrical system was constantly surging, and my 70-year-old bowling equipment (still) doesn't work perfectly. The system that keeps your score is particularly interesting to me. It's the thing you watch during your game, but…
Life & fun · Jul 2026
- EElevators▲1,680
Life & fun · Jul 2026 · john.fun
- 1W18 Words▲1,160
Life & fun · Jul 2026 · 18words.com
- BA
Over the past few months, our team has been building more and more slidedecks using web frontend technologies with coding harnesses like Claude Code, but a common complaint is to make even small edits we need to edit the code either manually or via the harness. To avoid this loop, I ended up creating Bento, a single HTML file with everything you need in a slide tool including animations and shared editing. There's no install or cloud login, everything works offline. The default deck is around 560 KB and it doesn't need to fetch anything once you got it. Open it in a browser and then you can…
Dev tools · Jul 2026 · bento.page
- GG
A few days ago I found myself trying out GLM 5.2 and was really positively impressed. The capabilities and security I was getting from this LLM are similar to those I've gotten from models like Claude or GPT, and this really surprised me. But then I thought, "I wonder how it would work on a normal computer like mine," and above all, "I wonder if it would work without going into OOM on a computer like mine." So I started working with the help of agents to test this possibility. I started converting the model to int4, understanding MTP usage, and if possible implementing DSA for long context.…
AI · Jul 2026 · github.com
