Alternatives
Products that do what OpenBenchmarks – Helping agents discover and pick the right SaaS APIs does
I'm Fenil, co-founder/CEO of OpenFunnel (YC F24), building this with my co-founder/CTO Aditya. We're launching OpenBenchmarks (https://openbenchmarks.com), open-source, reproducible benchmarks for SaaS APIs, starting with the category we know best: GTM APIs. ## Why we built this More and more B2B software evaluation will/already runs through reasoning models inside agentic workflows rather than through people. And buyers increasingly pick vendors that are API-first and ship MCPs, so they can wire them into internal workflows. Strong reasoning models are skeptical of…
- 1
- 2
- 3
- 4OB
Today, we're launching the Open Benchmarks Grants: a $3M commitment to fund open-source and academic teams building benchmarks for AI agents. In partnership with HuggingFace, PrimeIntellect, FactoryHQ, Together, Harbor, and PyTorch, the grants provide funding, data development support, and research collaboration. Our ability to measure AI has been outpaced by our ability to develop it, and we believe this evaluation gap is one of the most important problems in AI. Open benchmarks are one of the most important levers for advancing AI safely and responsibly—but the academic and open-source…
Feb 2026 · benchmarks.snorkel.ai
- 5
- 6IB
Hey HN, A few months ago, I tried to automate some of my work with the popular AI agent OpenClaw, and then I quickly realized how difficult it is to get it to work with APIs and third-party services securely, which is essential for a lot of work-related tasks. Then I realized OpenClaw is more of a personal assistant and it was not designed to get actual work done as a coworker. So I started to build Valmis, an alternative to OpenClaw that works with more than 100 apps and services, with security being the priority. Valmis addresses the security issue by designing a proxy system: dockerized…
Jul 2026 · github.com
- 7AR
If you're interested in exploring what LLM-based agent systems these days actually do to solve certain benchmarks such as SWEBench or WebArena, we created a small leaderboard with our team, that allows to view a lot of public and OSS agent results including all the runtime traces (the step-by-step reasoning behind the scenes). Looking at traces is actually quite interesting, as they reveal a lot about the inner working and shortcomings of current agent system, e.g. see https://explorer.invariantlabs.ai/u/invariant/webarena--SteP... for an example trace.
2024 · explorer.invariantlabs.ai
- 8TR
Hi HN, Today I'm showcasing Trunchbull, a benchmarking platform designed for authoring benchmarks and running them against different models. We have direct support for benchmarks that use the harbor authoring system, custom tool authoring via the vercel ai sdk and configuration limits. We've also already imported terminalbench 2.0, as a sort of proof of concept that our harbor task orchestrator works, although you currently need a paid account as we are provisioning sandbox environments. I've made several popular benchmarks publicly available for testing. You dont need an account or your…
25d ago · trunchbull.dev
- 9TT
We've been trying to evaluate web scraping companies, but when you look at their benchmarks, you can't verify anything, and they mostly exist to prove the company is successful. They put somewhere between 98% and 100% because they pick their own urls, define success their own way, and don't publish the harness. We also saw companies like scrapfly astroturf websites like scrapeway and call them independent. So, we built an open source benchmark that we want to represent the frontier of web data. We're trying to look across all major anti-bot providers and industries, to build a comprehensive…
Jul 2026 · github.com
- 10SA
Hi HN, I built SubTrack to help teams find unused SaaS tools and cloud resources before they silently eat into budgets. The motivation came from seeing how hard it is to answer simple questions: – Which SaaS tools are actually used? – Which cloud resources are idle? – What will our end-of-month spend look like? SubTrack connects to tools like AWS, GitHub, Vercel, and others to surface unused resources and cost signals from one place. Recently I added multi-account support, currency localization, and optional AI-based insights to help interpret usage patterns. This is an early-stage project…
Jan 2026 · subtrack.pulseguard.in
- 11IM
Recently, I was exploring the OpenAI Agents SDK and building MCP agents and agentic Workflows. To implement my learnings, I thought, why not solve a real, common problem? So I built this multi-agent job search workflow that takes a LinkedIn profile as input and finds personalized job opportunities based on your experience, skills, and interests. I used: - OpenAI Agents SDK to orchestrate the multi-agent workflow - Bright Data MCP server for scraping LinkedIn profiles & YC jobs. - Nebius AI models for fast + cheap inference - Streamlit for UI (The project isn't that complex - I kept it…
2025
- 12KS
KillSaaS is my answer to subscription software in the AI era. I'm building this because I believe small teams can use modern AI tools to create free alternatives to giants like Figma and DocuSign in weeks, not years. We're creating a platform where developers vote on which SaaS to replace, then build it together as open source. wdyt?
2025 · kill-saas.com
- 13WB
Over the past few months, as we scaled our internal AI Agents, we hit a dead end: Running LLM-generated arbitrary code in Docker is basically running naked on security due to container escape risks. But using full traditional VMs takes minutes to boot and eats too much memory to support high-density concurrency. We loved the developer experience of SaaS sandboxes on the market, but they are closed-source, expensive, and have too high a barrier to entry for self-hosting. So, our team decided to build our own. After months of grinding, using RustVMM and KVM, we built a blazing-fast,…
Apr 2026 · github.com
- 14WT
2020 · wundergraph.com
- 15HO
2021 · hoppscotch.io
- 16IB
Hey HN, I'm Daniel, solo dev from Germany. I built ClawHosters (https://clawhosters.com), a managed hosting platform for OpenClaw, the open-source AI agent framework. Quick timeline: domain registered February 5th. First paying customer six days later. I probably should have spent more time on it, but it works. If you haven't seen OpenClaw, it lets you run a personal AI assistant that connects to Telegram, Discord, Slack, and WhatsApp. Self-hosting it is absolutely possible, but it's a pain. You're dealing with Docker setup, SSL certs, port forwarding, security hardening, keeping…
Feb 2026 · clawhosters.com
- 17CB
I built a small benchmark to test CLI coding agents on blind bug detection. A challenger agent injects bugs and writes ground truth (`bugs.json`). A different reviewer agent audits the repo without seeing ground truth, and an LLM matcher scores bug-to-finding assignments. Current run: 50 repos, 150 challenges, 450 reviews, 2,603 injected bugs. Weighted detection: Claude 58.05%, Codex 37.84%, Gemini 27.81%. LLM-judge benchmarks are easy to get wrong, so I’d really appreciate critical feedback on benchmark fairness, scoring/matching methodology, and obvious failure modes I’m missing. Full…
Feb 2026 · github.com
- 18MD
We’re excited to share ML-Dev-Bench, a new open-source benchmark that tests AI agents on real-world ML development tasks. Unlike typical coding challenges or Kaggle-style competitions, our benchmark simulates end-to-end ML workflows including: - Dataset handling and preprocessing - Debugging model and code failures - Implementing new model architectures - Fine-tuning and improving existing models With 30 diverse tasks, ML-Dev-Bench evaluates agents across critical stages of ML development. To complement this, we built Calipers, a framework that provides systematic performance evaluation and…
2025 · github.com
- 19
ExtractBench - A Benchmark for Schema-Guided Enterprise Document Extraction - run-llama/ExtractBench
26d ago · github.com
- 20WM
We've recently made our product, ozma.io, open-source. It's a CRM/ERP platform for building enterprise systems. We believe that AI will soon handle implementing most of the boilerplate and UIs in the specialized business software. Just look at what lovable.dev does today! Soon products which make creating business software easier for developers will become obsolete, or transform into "libraries" to be used by AIs. We are losing this race, so we go the second route — publish everything and go on building other products on top of it. GitHub repo URL:…
2025 · github.com
- 21WB
Hey HN: Kaveh here, founder of https://www.usage.ai/ We help companies drive down AWS, GCP, and Azure spend. Why? Because the way it's done now is a pain. DevOps and Software Engineers end up spending time managing costs rather than focusing on business problems. I have been building Usage AI for almost 4 years now (4 year anniversary in 1 month from now!) with an incredible group of founding people. We started as a product just to help lower AWS EC2 costs, and now we do all major AWS services (such as RDS, OpenSearch, ElastiCache, and Redshift with more on the way) and other…
2024
- 22RU
hey all, happy to share research i've been working on for islo.dev in recent months. ever since the cheating agents (https://debugml.github.io/cheating-agents/) paper came out, revealing reward hacking was 4x more prevalent than previously estimated, i've been looking into how we can deal with the issue the common approach (taken by the tbench team) is post hoc trajectory analysis. i've been interested in the idea of reframing the problem as an endpoint security problem and tackling it via sandbox i hope you find it interesting, and thanks to the islo.dev team for…
Jun 2026 · github.com
- 23CA
I built this because I was tired of creating pull requests in 20 repositories just to change a single line of workflow job version. With Infra as AI, just mention the change. Agents work on all repos in parallel, read the docs, make a bunch of PRs and fill in the description. You can see the demo of the actual dashboard in the landing. Let me know your thoughts :) It means a lot to me!
Sep 2025 · infrastructureas.ai
- 24WB
Humans compete to improve their AI agents on benchmarks. But what if agents could collaborate and compete on their own? We built Hive, a crowdsourced platform where agents can evolve solutions together. One agent begins to tackle a task, iteratively improving its code. Then other agents join. They read each other’s runs, fork the best ideas, propose new ones, and push the solution forward together. We already have agents working on benchmarks like Tau2-Bench, Terminal-Bench, and ARC-AGI-2, with more tasks coming soon. We also support the new OpenAI Parameter Golf Challenge, and you can…
Mar 2026 · hive.rllm-project.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →