Composo – Evaluating and improving performance of LLM applications
Hey HN! We’re been building Composo - a platform that helps teams achieve high performance, guarantee accuracy & minimise the cost of LLM applications. Problem we’re solving: LLM applications are non-deterministic, so evaluating whether results are good or bad is highly subjective and often requires domain expertise. Iterating over 1000s of combinations of prompts, models, temperatures, RAG settings (& many other elements) is therefore very manual & time consuming. How we are solving it: Composo links directly to your application (in a simple to set up, but highly powerful way) which enables…
Composo – Evaluating and improving performance of LLM applications launched on January 5, 2024 with 30 votes, #923 of 4,437 launches that month and more than 75% of that year's launches. The site was still online in September 2026, 2 years on.
In plain words
Composo is a platform that helps teams optimize large language model applications by streamlining performance evaluation and testing. It addresses the challenge of LLM non-determinism by allowing users to test thousands of prompt, model, and parameter combinations without manual iteration. The platform integrates directly into applications and provides a remote control interface that enables both technical and non-technical team members to experiment with different configurations and measure accuracy while managing costs.
written from the facts on this page · September 2026
From the sources
Expert judgement, turned into signal: evaluation for your AI agents in production, and benchmarks and training signal for frontier models. Published research, open benchmarks.
The signal to evaluate your AI agents - and to benchmark and train the models themselves. Predefined rubrics don't learn and miss case-specific nuance. Make it accurate and it costs too much to run on everything. Our judge paper is accepted at two ICML 2026 workshops - RewardBench 2, 71.7 to 85.8. A generic judge is a weekend's work. One your experts would stand behind is three years of ours. A calibrated judge checks every output before it reaches a user, and holds back the ones that fail. A score on everything production runs, sharpening every time one of your experts corrects it. Failures group themselves into named modes, surfacing the risks nobody thought to look for. The cases worth…from composo.ai
In the maker’s words, at launch
Hey HN! We’re been building Composo - a platform that helps teams achieve high performance, guarantee accuracy & minimise the cost of LLM applications. Problem we’re solving: LLM applications are non-deterministic, so evaluating whether results are good or bad is highly subjective and often requires domain expertise.…
Does a similar job
all alternatives →
TensorZero2025 · github.com · ▲307Hi HN! We're Gabriel & Viraj, and we're excited to open source TensorZero. To be a little cheeky, TensorZero is an open-source platform that helps LLM applications graduate from API wrappers into defensible AI products. 1. Integrate our model gateway 2. Send metrics or feedback 3. Unlock compounding improvements in quality, cost, and latency It enables a data & learning flywheel for LLMs by unifying: • Inference: one API for all LLMs, with <1ms P99 overhead • Observability: inference & feedback → your database • Optimization: better prompts, models, inference strategies • Experimentation:…
Finetune LLaMA-7B on commodity GPUs using your own text2023 · github.com · ▲449I've been playing around with https://github.com/zphang/minimal-llama/ and https://github.com/tloen/alpaca-lora/blob/main/finetune.py, and wanted to create a simple UI where you can just paste text, tweak the parameters, and finetune the model quickly using a modern GPU. To prepare the data, simply separate your text with two blank lines. There's an inference tab, so you can test how the tuned model behaves. This is my first foray into the world of LLM finetuning, Python, Torch, Transformers, LoRA, PEFT, and Gradio. Enjoy!
Improve LLM Performance by Maximizing Iterative Development2024 · github.com · ▲104I have been working in AI space for a while now, first at FAANG with ML since 2021, then with LLM in start-ups since early 2023. I think LLM Application development is extremely iterative, more so than any other types of development. This is because to improve an LLM application performance (accuracy, hallucinations, latency, cost), you need to try various combinations of LLM models, prompt templates (e.g., few-shot, chain-of-thought), prompt context with different RAG architecture, different agent architecture, and more. There are thousands of possible combinations and you need a process…
- UDUnify – Dynamic LLM Benchmarks and SSO for Multi-Vendor Deployment2024 · ▲91
Hey HN! I’m the founder of Unify, and we’ve just released our Model Hub, which provides a collection of LLM endpoints with live runtime benchmarks all plotted across time: https://unify.ai/hub A key finding is that static tabular runtime benchmarks for LLMs simply do not work. It’s necessary to take a time-series perspective, and plot the variations through time. We currently have 21 models provided by: Anyscale, Perplexity AI, Replicate, Together AI, OctoAI, Mistral AI and OpenAI, with more on the roadmap. We test across different regions (Asia, US, Europe), with varied…
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 21d ago · stillwet.art
PixVerse R2▲381A real-time world model you can explore and change
AI · 30d ago · world.pixverse.video
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, January 2024
the whole month →
Amie▲1,334Turn meeting notes into automated workflows. Handle meetings, summaries, todos and emails (soon) with your AI personal assistant.
Work · 2024 · amie.so
- IM
Sit in Shade - Discover the optimal seat to stay in the shade during your journey. Sit In Shade provides real-time sun exposure data and route visualization.
Life & fun · 2024 · sitinshade.com
