Alternatives
Products that do what Benchscope does
Benchmark any LLM endpoint for your workload
- 1AT
I recently built a small open-source tool to benchmark different LLM API endpoints — including OpenAI, Claude, and self-hosted models (like llama.cpp). It runs a configurable number of test requests and reports two key metrics: • First-token latency (ms): How long it takes for the first token to appear • Output speed (tokens/sec): Overall output fluency Demo: https://llmapitest.com/ Code: https://github.com/qjr87/llm-api-test The goal is to provide a simple, visual, and reproducible way to evaluate performance across different LLM providers, including…
2025 · llmapitest.com
- 2
- 3FT
May 2026 · github.com
- 4LL
Hey Folks! I've been building an open source benchmark for measuring local LLM performance on your own hardware. The benchmarking tool is a CLI written on top of Llamafile to allow for portability across different hardware setups and operating systems. The website is a database of results from the benchmark, allowing you to explore the performance of different models and hardware configurations. Please give it a try! Any feedback and contribution is much appreciated. I'd love for this to serve as a helpful resource for the local AI community. For more check out: - Website:…
2025 · localscore.ai
- 5UD
Hey HN! I’m the founder of Unify, and we’ve just released our Model Hub, which provides a collection of LLM endpoints with live runtime benchmarks all plotted across time: https://unify.ai/hub A key finding is that static tabular runtime benchmarks for LLMs simply do not work. It’s necessary to take a time-series perspective, and plot the variations through time. We currently have 21 models provided by: Anyscale, Perplexity AI, Replicate, Together AI, OctoAI, Mistral AI and OpenAI, with more on the roadmap. We test across different regions (Asia, US, Europe), with varied…
2024
- 6
- 7
Compare LLMs on your data, measure, and pick the best.
Apr 2026 · trismik.com
- 8OO
Hey HN, Nir, Gal and Tomer here. We’re open-sourcing a set of extensions we’ve built on top of OpenTelemetry that provide visibility into LLM applications - whether it be prompts, vector DBs and more. Here’s the repo: https://github.com/traceloop/openllmetry. There’s already a decent number of tools for LLM observability, some open-source and some not. But what we found was missing for all of them is that they were closed-protocol by design, vendor-locking you to use their observability platform or their proprietary framework for running your LLMs. It’s still early in the…
2023 · github.com
- 9FL
Hi HN community, I have been working on benchmarking publicly available LLMs these past couple of weeks. More precisely, I am interested on the finetuning piece since a lot of businesses are starting to entertain the idea of self-hosting LLMs trained on their proprietary data rather than relying on third party APIs. To this point, I am tracking the following 4 pillars of evaluation that businesses are typically look into: - Performance - Time to train an LLM - Cost to train an LLM - Inference (throughput / latency / cost per token) For each LLM, my aim is to benchmark them for…
2023 · github.com
- 10
- 11
- 12DB
I built a CLI to benchmark DNS resolvers after discovering DNS was adding 300ms to my API requests. v0.3.0 just released with new features: compare: Test single domain across all resolvers top: Rank resolvers by latency/reliability/balanced monitor: Continuous tracking with threshold alerts 1,400+ downloads in first week. Quick start: pip install dns-benchmark-tool dns-benchmark compare --domain google.com CLI stays free forever. Hosted version (multi-region, historical tracking, alerts) coming Q1 2026. GitHub: https://github.com/frankovo/dns-benchmark-tool…
Nov 2025 · github.com
- 13
- 14YD
If you've built any web-based app in the last 15 years, you probably used something like Datadog, New Relic, Sentry, etc. to monitor and trace your app, right? Why should it be different when the app you're building happens to be using LLMs? So today we're open-sourcing OpenLLMetry-JS. It's an open protocol and SDK, based on OpenTelemetry, that provides traces and metrics for LLM JS/TS applications and can be connected to any of the 15+ tools that already support OpenTelemetry. Here's the repo: https://github.com/traceloop/openllmetry-js A few months ago we launched…
2024 · github.com
- 15
- 16
- 17
Aggregate uptime monitoring across OpenAI, Claude, and more
Apr 2026 · tools.lamatic.ai
- 18
- 19AB
2025 · github.com
- 20
- 21OS
Looking for the cheapest place to deploy llama 3.1 model? Don't worry we have found it so you don't have to.
2024 · github.com
- 22
- 23
- 24
Ranked by how close each launch is in meaning, then by votes. Refine with a description →