Faster LLM evaluation with Bayesian optimization
Recently I've been working on making LLM evaluations fast by using bayesian optimization to select a sensible subset. Bayesian optimization is used because it’s good for exploration / exploitation of expensive black box (paraphrase, LLM). I would love to hear your thoughts and suggestions on this!
Faster LLM evaluation with Bayesian optimization launched on February 13, 2024 with 131 votes, #382 of 3,962 launches that month and more than 95% of that year's launches. The site was still online in September 2026, 2 years on.
In plain words
This tool speeds up large language model evaluations by using Bayesian optimization to intelligently select a representative subset of test cases to run. Rather than testing every scenario, it balances exploring new cases with exploiting known good ones, reducing the cost and time of expensive LLM evaluation processes. The tool is designed for developers and researchers who need faster feedback when testing language models.
written from the facts on this page · September 2026
From the sources
Bayesian Optimization as a Coverage Tool for Evaluating LLMs. Accurate evaluation (benchmarking) that's 10 times faster with just a few lines of modular code. - rentruewang/bocoel
Does a similar job
all alternatives →
Speeding up LLM inference 2x times (possibly)2024 · asciinema.org · ▲419Here's a project I've been working on for the last few months. It's a new (I think) algorithm, that allows to adjust smoothly - and in real time - how many calculations you'd like to do during inference of an LLM model. It seems that it's possible to do just 20-25% of weight multiplications instead of all of them, and still get good inference results. I implemented it to run on M1/M2/M3 GPU. The mmul approximation itself can be pushed to run 2x fast before the quality of output collapses. The inference speed is just a bit faster than Llama.cpp's, because the rest of implementation…
Improve LLM Performance by Maximizing Iterative Development2024 · github.com · ▲104I have been working in AI space for a while now, first at FAANG with ML since 2021, then with LLM in start-ups since early 2023. I think LLM Application development is extremely iterative, more so than any other types of development. This is because to improve an LLM application performance (accuracy, hallucinations, latency, cost), you need to try various combinations of LLM models, prompt templates (e.g., few-shot, chain-of-thought), prompt context with different RAG architecture, different agent architecture, and more. There are thousands of possible combinations and you need a process…
Find the best local LLM for your hardware, ranked by benchmarksMay 2026 · github.com · ▲283Find the local LLM that actually runs and performs best on your hardware. Ranked by real, recency-aware benchmarks, not parameter count. One command, run it instantly. - Andyyyy64/whichllm
- ABAutoThink – Boosts local LLM performance with adaptive reasoning2025 · ▲397
I built AutoThink, a technique that makes local LLMs reason more efficiently by adaptively allocating computational resources based on query complexity. The core idea: instead of giving every query the same "thinking time," classify queries as HIGH or LOW complexity and allocate thinking tokens accordingly. Complex reasoning gets 70-90% of tokens, simple queries get 20-40%. I also implemented steering vectors derived from Pivotal Token Search (originally from Microsoft's Phi-4 paper) that guide the model's reasoning patterns during generation. These vectors encourage behaviors like numerical…
A radically more scalable alternative to Bayesian Optimization2018 · github.com · ▲96Experimental Global Optimization Algorithm. Contribute to chrisstroemel/Simple development by creating an account on GitHub.
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 20d ago · stillwet.art
PixVerse R2▲381A real-time world model you can explore and change
AI · 29d ago · world.pixverse.video
Source: Show HN post ↗
Launched alongside, February 2024
the whole month →
MarketingIdeas.com▲1,230Never run out of marketing ideas ever again
Growth · 2024 · marketingideas.com
