Continuous-eval – Granular evaluation of GenAI pipelines
Hi HN - we are the creators of “continuous-eval”, an open-source tool to test and evaluate generative AI apps. "Continuous-eval" came from our efforts to measure, validate and improve the reliability of a finance AI copilot we were developing for banks. End-to-end evaluation was not enough for us. We wanted to have granular evaluations that help pinpoint the bottlenecks and identify what / how to improve. We’ve since developed more metrics and made the framework more flexible so it can evaluate components like agent tool use, code change, retrieval steps, etc. Let us know what you think…
Continuous-eval – Granular evaluation of GenAI pipelines launched on February 25, 2024 with 10 votes, #1,321 of 3,962 launches that month and more than 75% of that year's launches. The site was still online in September 2026, 2 years on.
In plain words
Continuous-eval is an open-source testing framework for evaluating generative AI applications at a granular level. It enables developers to assess individual components such as agent tool use, code changes, and retrieval steps rather than relying solely on end-to-end evaluation. The tool was created to help identify performance bottlenecks and guide improvements in AI systems, originally developed for finance applications. It is designed for teams building and refining generative AI products.
written from the facts on this page · September 2026
From the sources
Data-Driven Evaluation for LLM-Powered Applications - relari-ai/continuous-eval
In the maker’s words, at launch
Hi HN - we are the creators of “continuous-eval”, an open-source tool to test and evaluate generative AI apps. "Continuous-eval" came from our efforts to measure, validate and improve the reliability of a finance AI copilot we were developing for banks. End-to-end evaluation was not enough for us. We wanted to have…
Does a similar job
all alternatives →
Gentrace – evaluation and observability for generative AI2023 · gentrace.ai · ▲67Hi HN, Gentrace is our new evaluation and observability tool for generative AI (open beta). Generative pipelines are hard to evaluate because outputs are subjective. Lots of developers end up just doing “gut checks” on a few inputs before shipping changes, or they build up a spreadsheet of test cases that they manually run through the pipeline. Some companies outsource filling out the spreadsheet. However, in any of these cases, you end up with a very slow and expensive process for evaluation. At one point, we did this too. Gentrace is the result of a pivot; it was an internal tool we used…
git-lrcFeb 2026 · hexmos.com · ▲300Hi HN, I'm the author of git-lrc, would appreciate some feedback from the community Last year my team started using AI coding tools more heavily, and we found ourselves generating tons of code, but spending less time looking at the stuff that's been generated. We felt like we were losing a bit of grip/understanding of what we were building. Regressions occasionally slipped through. Sometimes changes made it all the way to production only to be reverted later. We tried several AI code review tools, but most operate at PR time. That felt too late. I wanted review to happen while the…
Agent-skills-eval – Test whether Agent Skills improve outputsMay 2026 · github.com · ▲79A test runner for agentskills.io-style AI agent skills - darkrishabh/agent-skills-eval
Create LLM graders and run evals in JavaScript with one file2025 · github.com · ▲28Hi HN! Run it: OPENROUTER_API_KEY="sk" npx bff-eval --demo We built a tool to help people take LLM outputs and easily grade them / eval them to know how good an assistant response is. We've built a number of LLM apps, and while we could ship decent tech demos, we were disappointed with how they'd perform over time. We worked with a few companies who had the same problem, and found out scientifically building prompts and evals is far from a solved problem... writing these things feels more like directing a play than coding. Inspired by Anthropic's constitutional ai concepts, and amazing…
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 21d ago · stillwet.art
PixVerse R2▲381A real-time world model you can explore and change
AI · 30d ago · world.pixverse.video
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, February 2024
the whole month →
MarketingIdeas.com▲1,230Never run out of marketing ideas ever again
Growth · 2024 · marketingideas.com
