A tool to benchmark LLM APIs (OpenAI, Claude, local/self-hosted)
I recently built a small open-source tool to benchmark different LLM API endpoints — including OpenAI, Claude, and self-hosted models (like llama.cpp). It runs a configurable number of test requests and reports two key metrics: • First-token latency (ms): How long it takes for the first token to appear • Output speed (tokens/sec): Overall output fluency Demo: https://llmapitest.com/ Code: https://github.com/qjr87/llm-api-test The goal is to provide a simple, visual, and reproducible way to evaluate performance across different LLM providers, including…
A tool to benchmark LLM APIs (OpenAI, Claude, local/self-hosted) launched on June 29, 2025 with 55 votes, #532 of 7,999 launches that month and more than 90% of that year's launches. The site was still online in September 2026, 1 year on.
In plain words
This open-source tool benchmarks different LLM APIs including OpenAI, Claude, and self-hosted models like llama.cpp. It runs configurable test requests and measures two key performance metrics: first-token latency in milliseconds and output speed in tokens per second. The tool supports OpenAI-compatible APIs, Anthropic's Claude, and local endpoints, with results presented visually for easy comparison across providers. It can be self-hosted using Docker Compose, making it useful for developers evaluating LLM performance across multiple services.
written from the facts on this page · September 2026
From the sources
llm api test - Compare and analyze AI model performance
Compare performance across multiple AI model API providers and models with comprehensive testing andfrom llmapitest.com
In the maker’s words, at launch
I recently built a small open-source tool to benchmark different LLM API endpoints — including OpenAI, Claude, and self-hosted models (like llama.cpp). It runs a configurable number of test requests and reports two key metrics: • First-token latency (ms): How long it takes for the first token to appear • Output speed…
Does a similar job
all alternatives →
LLM StatsOct 2025 · llm-stats.com · ▲293I was overspending on GPT-4o. It was really hard to compare different models I could switch to, so I built this LLM comparison tool. It shows leaderboards, pricing, and performance data across 100+ LLMs (including all major providers and open-source models). Key features: - Live pricing comparisons - Benchmark Scores (MMLU, HumanEval, GPQA, etc.) - Context length vs cost analysis - Speed/throughput tests across providers - Quality vs price visualizations - Open source (all data verifiable) Try it out: https://llmstats.com I'd like to know your opinion :) Tech stack: Next.js,…
LocalScore – Local LLM Benchmark2025 · localscore.ai · ▲124Hey Folks! I've been building an open source benchmark for measuring local LLM performance on your own hardware. The benchmarking tool is a CLI written on top of Llamafile to allow for portability across different hardware setups and operating systems. The website is a database of results from the benchmark, allowing you to explore the performance of different models and hardware configurations. Please give it a try! Any feedback and contribution is much appreciated. I'd love for this to serve as a helpful resource for the local AI community. For more check out: - Website:…
LLM Gateway2025 · llmgateway.io · ▲296I built an OpenAI-compatible LLM gateway that routes requests to OpenAI, Anthropic, Ollama, vLLM, llama-server, SGLang... anything that speaks /v1/chat/completions. Single Go binary, one YAML config file, no infrastructure. It does the things you'd expect from this kind of gateway... semantic routing via a three-layer cascade (keyword heuristics, embedding similarity, LLM classifier) that picks the best model when clients omit the model field, weighted round-robin load balancing across local inference servers with health checks and failover. The part I think is most…
- UDUnify – Dynamic LLM Benchmarks and SSO for Multi-Vendor Deployment2024 · ▲91
Hey HN! I’m the founder of Unify, and we’ve just released our Model Hub, which provides a collection of LLM endpoints with live runtime benchmarks all plotted across time: https://unify.ai/hub A key finding is that static tabular runtime benchmarks for LLMs simply do not work. It’s necessary to take a time-series perspective, and plot the variations through time. We currently have 21 models provided by: Anyscale, Perplexity AI, Replicate, Together AI, OctoAI, Mistral AI and OpenAI, with more on the roadmap. We test across different regions (Asia, US, Europe), with varied…
finetune LLMs via the Finetuning Hub2023 · github.com · ▲80Hi HN community, I have been working on benchmarking publicly available LLMs these past couple of weeks. More precisely, I am interested on the finetuning piece since a lot of businesses are starting to entertain the idea of self-hosting LLMs trained on their proprietary data rather than relying on third party APIs. To this point, I am tracking the following 4 pillars of evaluation that businesses are typically look into: - Performance - Time to train an LLM - Cost to train an LLM - Inference (throughput / latency / cost per token) For each LLM, my aim is to benchmark them for…
Questions this answers
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 22d ago · stillwet.art
FastRouter.ai▲364Route requests to the right LLM for cost, latency & quality
AI · 23d ago · fastrouter.ai
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, June 2025
the whole month →- IM
Hey HN! Pilots everywhere are required to keep a logbook of all their flying hours, aircraft, airports, and so on. Since I track everything digitally (some people still just use paper logbooks!), I put together some data visualizations and a few 3D globes to show my flying history. This globe is probably my favourite so far: https://jameshard.ing/pilot/globes/all If you’ve got ideas for other graphs or ways to show this kind of data, I’d love to hear them!
Life & fun · 2025 · jameshard.ing
Stunning presentations with AI. No design skills required.
AI · 2025 · chroniclehq.com
Build native mobile apps for iOS and Android without code.
Dev tools · 2025 · bble.io
All your connections, across all your socials.
AI · 2025 · pally.com
I was the main contributor to workout.lol, an open-source fitness app to easily build a workout routine. The project had traction (1.4k GitHub stars, 95 forks, ~20K visits/month), but was eventually sold due to video licensing hurdles. The new owner stopped maintaining it, and the repo went abandoned. Over the next 9 months, I sent 15 emails to try to save it : no replies. Feature requests & issues were ignored. The community was left with a "broken" tool let's say. I couldn't just let it die So I built the new version from scratch with the same open-source spirit, but a better…
Dev tools · 2025 · github.com