CAFE — Compound-AI Factorial Evaluation
Stop guessing which AI config is better. Prove it.
CAFE — Compound-AI Factorial Evaluation launched on July 14, 2026 with 6 votes, #3,356 of 24,623 launches that month and more than 75% of that year's launches.
What it does
CAFE treats every knob in your AI pipeline - retrieval, reranking, prompts, models, and tools - as an experimental factor. It runs factorial experiments, evaluates outputs using a configurable LLM (and optionally human reviewers), and applies mixed-effects models to determine: - Which techniques actually improve quality - How much each technique contributes - Whether the observed differences are statistically significant Open source and self-hostable.
A design-of-experiments platform for evaluating compound AI systems. Stop guessing which config is better. Prove it.
CAFE is a design-of-experiments platform for RAG, agents, and LLM chains. Declare the factors you want to vary, run them with replication, and attribute quality to the components that actually drive it — with p-values. [ FACTORIAL ANALYSIS COMPLETE ] Score | 0.95 | █ 0.78 | █ █ 0.55 | █ █ █ 0.30 | █ █ █ █ +------------ none dense dense+ rerank Built with You bring your pipeline as a black box; CAFE builds the experiment around it and tells you which components actually drive quality — and which just add cost. Declare categorical, ordinal, or continuous factors, and CAFE expands the full factorial design automatically — or a fractional 2 k-p design when the full grid is too large, with…from cafe-ai.de
Does a similar job
all alternatives →- FLFactorio Learning Environment – Agents Build Factories2025 · jackhopkins.github.io · ▲749
I'm Jack, and I'm excited to share a project that has channeled my Factorio addiction recently: the Factorio Learning Environment (FLE). FLE is an open-source framework for developing and evaluating LLM agents in Factorio. It provides a controlled environment where AI models can attempt complex automation, resource management, and optimisation tasks in a grounded world with meaningful constraints. A critical advantage of Factorio as a benchmark is its unbounded nature. Unlike many evals that are quickly saturated by newer models, Factorio's geometric complexity scaling means it won't be…
- EFEcho – Fable-level results at 1/3 the cost using open-weight modelsJul 2026 · ▲484
I’ve been building Echo (https://echo.tracerml.ai/), an experiment in making one AI system out of a pool of open-weight models rather than choosing a single model and using it for every task. It started with a simple experiment. I took a group of models, including GLM-5.2, Kimi K2.7 and others, and ran them on the same evaluations. Then I measured what would happen if, for each problem, you somehow knew in advance which models would be useful and how their outputs should be combined. That hypothetical system performed substantially better than any individual model in the pool.…
TensorZero2025 · github.com · ▲307Hi HN! We're Gabriel & Viraj, and we're excited to open source TensorZero. To be a little cheeky, TensorZero is an open-source platform that helps LLM applications graduate from API wrappers into defensible AI products. 1. Integrate our model gateway 2. Send metrics or feedback 3. Unlock compounding improvements in quality, cost, and latency It enables a data & learning flywheel for LLMs by unifying: • Inference: one API for all LLMs, with <1ms P99 overhead • Observability: inference & feedback → your database • Optimization: better prompts, models, inference strategies • Experimentation:…
- ADA Digital Twin of my coffee roaster that runs in the browser2025 · autoroaster.com · ▲157
I built this website to host a data-driven model of my coffee sample roaster. I realized after 20 or so batches on the machine that while the controls are intuitive (heat, fan, and drum speeds), the physics can be unintuitive. I wanted to use my historical roast data to create and tune a model that I could use to do roast planning, control, and to help me build my own intuition for roasting. This website lets you interact with my roaster in a virtual, risk-free setting! The models are custom Machine Learning modules that honor roaster physics and bean physics (this is not…
We built an AI to review your pull requests2025 · infinitcode.ai · ▲53Hey HN, We’re two developers (co-founders) with a team of 20 who got tired of spending hours reviewing PRs, so we built Infinitcode.ai, an AI-powered code reviewer that: - *Summarizes PRs in plain English*: No more deciphering 1,000-line diff jungles - *Catches more than bugs*: Security holes, performance pitfalls, code smells, even typos (yes, we’ll flag “vurnerabilities” and vulnerabilities) - *Zero onboarding*: Works instantly—no “let me learn your codebase for weeks” nonsense. Why we’re posting: We’re in alpha and need brutal honesty. Roast our tool, mock our UI, or tell us why AI will…
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 20d ago · stillwet.art
PixVerse R2▲381A real-time world model you can explore and change
AI · 29d ago · world.pixverse.video
Source: Product Hunt launch ↗
Launched alongside, July 2026
the whole month →- IR
I might be the only SRE on Earth with his own bowling center. It's a more in-depth gig than you'd think. My family and I bought an abandoned 8-lane bowling center in the rural mid-west. In our small town there weren't many recreation options for families. You've heard of a food desert? This is an R&R desert. It had been abandoned for a good reason. The roof leaks, the electrical system was constantly surging, and my 70-year-old bowling equipment (still) doesn't work perfectly. The system that keeps your score is particularly interesting to me. It's the thing you watch during your game, but…
Life & fun · Jul 2026
18 Words▲1,160Play 18 Words, the free daily word puzzle. The official home is 18words.com. Tap Play now to open today
Life & fun · Jul 2026 · 18words.com
- BA
Over the past few months, our team has been building more and more slidedecks using web frontend technologies with coding harnesses like Claude Code, but a common complaint is to make even small edits we need to edit the code either manually or via the harness. To avoid this loop, I ended up creating Bento, a single HTML file with everything you need in a slide tool including animations and shared editing. There's no install or cloud login, everything works offline. The default deck is around 560 KB and it doesn't need to fetch anything once you got it. Open it in a browser and then you can…
Dev tools · Jul 2026 · bento.page
A few days ago I found myself trying out GLM 5.2 and was really positively impressed. The capabilities and security I was getting from this LLM are similar to those I've gotten from models like Claude or GPT, and this really surprised me. But then I thought, "I wonder how it would work on a normal computer like mine," and above all, "I wonder if it would work without going into OOM on a computer like mine." So I started working with the help of agents to test this possibility. I started converting the model to int4, understanding MTP usage, and if possible implementing DSA for long context.…
AI · Jul 2026 · github.com
OpenSEO▲928Hello! For the last 2 months, I've been building OpenSEO. It's a user friendly UI wrapping the DataForSEO API which has high quality SEO data for keywords, backlinks, AI SEO and competitor research. Instead of paying $100+ per month, you buy DataForSEO credits and pay only for usage. The coolest thing about the project so far has been hearing about what people are building on top of it. People are using coding agents to quickly integrate with Google Search Console, Google Ads and building their own AI SEO and reporting features. Many were already building their own custom solutions around…
Dev tools · Jul 2026 · openseo.so