Composo – Evaluating and improving performance of LLM applications
Hey HN! We’re been building Composo - a platform that helps teams achieve high performance, guarantee accuracy & minimise the cost of LLM applications. Problem we’re solving: LLM applications are non-deterministic, so evaluating whether results are good or bad is highly subjective and often requires domain expertise. Iterating over 1000s of combinations of prompts, models, temperatures, RAG settings (& many other elements) is therefore very manual & time consuming. How we are solving it: Composo links directly to your application (in a simple to set up, but highly powerful way) which enables…
In plain words
Composo is a platform that helps teams optimize large language model applications by streamlining performance evaluation and testing. It addresses the challenge of LLM non-determinism by allowing users to test thousands of prompt, model, and parameter combinations without manual iteration. The platform integrates directly into applications and provides a remote control interface that enables both technical and non-technical team members to experiment with different configurations and measure accuracy while managing costs.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
Hey HN! We’re been building Composo - a platform that helps teams achieve high performance, guarantee accuracy & minimise the cost of LLM applications. Problem we’re solving: LLM applications are non-deterministic, so evaluating whether results are good or bad is highly subjective and often requires domain expertise. Iterating over 1000s of combinations of prompts, models, temperatures, RAG settings (& many other elements) is therefore very manual & time consuming. How we are solving it: Composo links directly to your application (in a simple to set up, but highly powerful way) which enables it to function like a remote control for your application. Once set up, anyone on your team (inc. non technical domain experts or PMs), can use Composo to easily test out your application with different models, prompts, temperatures & RAG settings (or any other variable in your codebase you decide to make available at initial set up). Crucially, this is simple enough to be used by anyone, but powerful enough for any application (e.g. real apps built in code using agents etc). This testing can be done in both our playground & our evaluation suite: 1) Playground: Here you can ‘chat’ with your application in a UI similar to the openAI playground, but with inputs being runs on your actual application rather than a simple LLM call & with the ability to change any variables you like directly within the Composo UI (e.g. system message, temperature, model, RAG settings). 2) Evaluation suite: Here you can conduct rigorous testing & evaluation on your application either ad-hoc while in development, or repeated over time to check for performance regression. Our test suite contains automated evaluation tools including: evaluation in comparison to ground truth answers (with exact match, vector similarity & LLM graded similarity), with specific criteria (e.g. code validity, JSON validity, specific keyword inclusion or exclusion) & AI grading (this uses the Composo AI critic which leverages the latest research in LLM auto-evaluation under the hood). The easiest things to get started with, without having to link an application or even sign up, are: 1) Play with different models in our playground by chatting directly or using our demo apps (e.g. an AI doctor) 2) Automate your prompt writing & optimisation with our AI prompt writer Thanks so much, and would be hugely grateful for any feedback!
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 26d ago · cactuscompute.com
Launched alongside, January 2024
the whole month →
- IM
Life & fun · 2024 · sitinshade.com
