Alternatives
Products that do what Randomly switching between LMs at every step boosts SWE-bench score does
What if your agent uses a different LM at every turn? We let mini-SWE-agent randomly switch between GPT-5 and Sonnet 4 and it scored higher on SWE-bench than with either model separately. GPT-5 by itself gets 65.0%, Sonnet 4 64.8%, but randomly switching at every step gets us 67.2% This result came pretty surprising to us. There's a few more experiments in the blog post.
- 1
- 2MS
2025 · github.com
- 3
- 4BA
I built CodeLens.AI - a tool that compares how 6 top LLMs (GPT-5, Claude Opus 4.1, Claude Sonnet 4.5, Grok 4, Gemini 2.5 Pro, o3) handle your actual code tasks. How it works: - Upload code + describe task (refactoring, security review, architecture, etc.) - All 6 models run in parallel (~2-5 min) - See side-by-side comparison with AI judge scores - Community votes on winners (blind voting) - Each evaluation gets reflected in the overall AI model leaderboard, showing us best ones Why I built this: Existing benchmarks (HumanEval, SWE-Bench) don't reflect real-world developer tasks. I wanted to…
Oct 2025 · codelens.ai
- 5WM
We wanted to test if a smaller model like GPT-4.1-mini could beat its bigger brother 4.1 at the game Tic-Tac-Toe using only context engineering. We put them in a 100-game tournament. For the smaller model, we gave it a few examples of winning moves from past games right before it made its own move. The results were clear. Without the examples, the smaller model struggled against GPT-4.1. With the examples, its effectiveness increased by nearly 200%, and it consistently won. It's a simple demonstration, but it shows that a smaller, faster model with good, timely examples can outperform a more…
2025 · github.com
- 6AR
If you're interested in exploring what LLM-based agent systems these days actually do to solve certain benchmarks such as SWEBench or WebArena, we created a small leaderboard with our team, that allows to view a lot of public and OSS agent results including all the runtime traces (the step-by-step reasoning behind the scenes). Looking at traces is actually quite interesting, as they reveal a lot about the inner working and shortcomings of current agent system, e.g. see https://explorer.invariantlabs.ai/u/invariant/webarena--SteP... for an example trace.
2024 · explorer.invariantlabs.ai
- 7LI
Hey HN! We built Lunon to make LLM development way less of a headache. Ever wanted to see how different models handle the same prompt without all the setup hassle? That's what we fixed. Our API lets you compare Claude, GPT, Mistral and others in real-time with just a few lines of code. No more complex infrastructure or managing multiple API connections - we handle all that boring stuff behind the scenes. Plus, you can cut costs by intelligently routing requests to the right model for each task. Use the powerful (expensive) models only when you really need them. If you're building with LLMs…
2025 · lunon.com
- 8
Coding agents craft arbitrary code so securing them is more complicated than red-teaming. We post trained a cyber-security small llm, changed how it reasons and supplemented our controls using program analysis techniques such as inline reference monitoring to outperform GPT5.5-xhigh on hard benchmarks like LinuxArena and SleightBench. Free product available at harden.run and full benchmarks in the blog post.
9d ago · harden.run
- 9MI
Hi HN! I lead product at Vectara and we've just released a new LLM in our platform that outperforms GPT4 and Gemini 1.5 Pro on RAG tasks. Vectara is a Retrieval Augmented Generation (RAG) platform primarily deployed as a SaaS service which includes a generous free tier so you can try it for free. The way we've been able to offer a "better but cheaper" is that we focus a lot of our attention on taking smaller models (which can be hosted in a cost efficient way) and fine tuning them to specific tasks: in this case RAG. This ends up with a model that is less capable of arbitrary tasks like…
2024 · vectara.com
- 10PP
I've been working on applying LLMs to long-context, verifiable problems over the past year, and today I'm releasing a benchmark of 62,000 pencil puzzles across 94 types (sudoku, nonori, slitherlink, etc.). The benchmark also allows for intermediate checks /rule breaks for all varieties at any step. I tested 51 models against a subset (300 puzzles) in two modes: single-shot (output the full solution) and agentic (iterate with verifier feedback). Some results: - Best model (GPT 5.2@xhigh) solves 56%. (~ half the puzzles are unsolved by any model) - Agentic solves average 29 turns. The…
Mar 2026 · ppbench.com
- 11HT
Fun little project, had Gemini 2.5 Pro summarize HN's top 30 each hour, both the stories and comment sections. Pretty impressed with Gemini 2.5. It's probably the first model other than Claude 3.7 Sonnet where I actually find the output readable. I normally use 3.7 Sonnet for coding, but used Gemini for the codegen on this one as well. Was pretty impressed! Using Cursor, it seemed to instruction-follow better than Claude generally does, and remain lucid during very long agent sessions. Thanks for your feedback!
2025 · tinysums.ai
- 12VE
I'm building a Telegram bot to practice Dutch. GPT-4o-mini kept picking vocabulary words I already knew, so I built a classical NLP pipeline to do it instead. It takes a short text + learner level (A0–B1) and returns the best words to study, using Stanza for parsing and corpus frequency ranks (SUBTLEX-NL, srLex, SUBTLEX-US) for scoring. Wins at A1/A2, loses at A0 where the LLM picks more obvious words. I also tried adding multi-word phrases (ADJ+NOUN, VERB+NOUN, phrasal verbs) backed by NPMI-scored collocation whitelists. Couldn't beat GPT there because it just "knows" which phrases…
Mar 2026 · huggingface.co
- 13CB
I built a small benchmark to test CLI coding agents on blind bug detection. A challenger agent injects bugs and writes ground truth (`bugs.json`). A different reviewer agent audits the repo without seeing ground truth, and an LLM matcher scores bug-to-finding assignments. Current run: 50 repos, 150 challenges, 450 reviews, 2,603 injected bugs. Weighted detection: Claude 58.05%, Codex 37.84%, Gemini 27.81%. LLM-judge benchmarks are easy to get wrong, so I’d really appreciate critical feedback on benchmark fairness, scoring/matching methodology, and obvious failure modes I’m missing. Full…
Feb 2026 · github.com
- 14GV
Hey HN, I just updated my project that compares some LLMs. It uses your prompt for all the models and runs at the same time. You can see the results being generated in real-time and decide what's the best for your use case. I'm open to any suggestions and feedback. Thanks!
2024 · geminivsgpt.com
- 15ML
Time to first token is 39% faster Agent wall times decrease by 46% No swaps Tracks your resource usage in real-time and adjusts how the model runs so that it works perfectly on your device. Implements KV cache sizing, prefix caching, live RAM pressure management, context trimming, KV quantization, and more. Built a ton of features
Jun 2026 · autotunellm.com
- 16SD
Hey HN! After spending way too many nights debugging flaky AI tests, I built SteadyText. It's a simple python library for deterministic llm generations and embeddings. We use it in production for: - Testing our AI features (zero flakes in 3 months) - CLI tools that need consistent outputs - Reproducible documentation examples It's not for creative tasks - this is specifically for when you need AI to be boring and predictable. Think of it as the opposite of ChatGPT. The coolest part? It includes a Postgres extension. You can now do: SELECT steadytext_generate('explain this query: ...'); And…
2025 · steadytext.julep.ai
- 17G5
TL;DR: we just made GPT-5 available for free on Gensee, a developer-oriented AI Agent optimization and deployment platform: https://platform.gensee.ai/ This is a crazy week with a bunch of model releases: gpt-oss, Claude-Opus-4.1, and now today's GPT-5. It may feel impossible for agent developers to keep up, with all the manual migrating, re-testing, and analyzing. We built Gensee to solve exactly this problem. Gensee lets you see the immediate impact of a new model on your already built agents and workflows. Here’s how it works: - Instant Model Swapping: Have an agent running…
2025 · gensee.ai
- 18SA
Hi HN, We’re building https://www.switchpoint.dev – a drop-in replacement for OpenAI’s API that reduces LLM cost by smartly routing across models (e.g., Claude, Gemini, GPT-4) depending on subject and difficulty of the task. Why we built this: LLM costs are spiraling—especially for products doing retrieval, agentic reasoning, or even just high-volume chat. We were frustrated with paying GPT-4 rates when most queries didn’t need it. So we built a router that: - Starts with cheaper/free models (like Llama 8B, 4o-mini, 2.0 flash) - Streams responses and upgrades on failure - Acts…
2025 · switchpoint.dev
- 19OS
Hey HN, I fine-tuned a small open-source model on golf forecasting and it beats GPT-5 at predicting golf outcomes. The same approach can be used to build a specialized model in any domain, you just need to update a few search queries. We fine-tuned gpt-oss-120b with LoRA on 3,178 golf forecasting questions, using GRPO with Brier score as the reward. Our model outperformed GPT-5 on Brier Skill (17% vs 12.8%) and ECE (6% vs 10.6%) on 855 held-out questions. How to try it: the model and dataset are open-source, with code, on Hugging Face. How to build your own specialized model: Update the…
Feb 2026 · huggingface.co
- 20AL
Try it out here: https://labs.refuel.ai/playground Refuel LLM (84.2%) outperforms trained human annotators (80.4%), GPT-3-5-turbo (81.3%), PaLM-2 (82.3%) and Claude (79.3%) across a benchmark of 15 text labeling datasets. It is a Llama-v2-13b base model, trained on over 2500 unique datasets (5.24B tokens) spanning categories such as classification, entity resolution, matching, reading comprehension and information extraction.
2023
- 21NL
Refuel LLM (84.2%) outperforms trained human annotators (80.4%), GPT-3-5-turbo (81.3%), PaLM-2 (82.3%) and Claude (79.3%) across a benchmark of 15 text labeling datasets. It is a Llama-v2-13b base model, trained on over 2500 unique datasets (5.24B tokens) spanning categories such as classification, entity resolution, matching, reading comprehension and information extraction. Here is the interactive demo: https://labs.refuel.ai/playground. Pretty fun to play with!
2023
- 22MD
We’re excited to share ML-Dev-Bench, a new open-source benchmark that tests AI agents on real-world ML development tasks. Unlike typical coding challenges or Kaggle-style competitions, our benchmark simulates end-to-end ML workflows including: - Dataset handling and preprocessing - Debugging model and code failures - Implementing new model architectures - Fine-tuning and improving existing models With 30 diverse tasks, ML-Dev-Bench evaluates agents across critical stages of ML development. To complement this, we built Calipers, a framework that provides systematic performance evaluation and…
2025 · github.com
- 23HW
Hello everyone! I’m thrilled to announce the latest feature from Mutahunter.ai, the ultimate tool for finding and fixing weaknesses in your code. We’ve designed Mutahunter to leverage mutation testing powered by advanced LLMs, helping you uncover vulnerabilities and enhance your code quality effortlessly. Introducing our newest feature: Detailed Mutation Testing Reports! After running our mutation tests, Mutahunter now generates comprehensive reports that clearly summarize: • Vulnerable code gaps • Test case gaps These reports significantly reduce the cognitive load on developers by…
2024 · github.com
- 24LO
Hi HN! I built LLM OneStop (https://www.llmonestop.com), a unified interface for accessing multiple AI language models in one place. The main problem I wanted to solve: constantly switching between different AI platforms, managing multiple subscriptions, and losing conversation context when comparing outputs across models. Key features: Switch between GPT-4, Claude, Gemini, Llama, and other models mid-conversation Compare responses side-by-side Single interface instead of juggling multiple tabs/subscriptions Free tier available to try it out (no credit card needed) "Connect"…
Nov 2025 · llmonestop.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →