AI · alternatives · 2026

24 alternatives to Evaluate LLM-based RAG Applications with automated test set generation

🐢 Open-Source Evaluation & Testing library for LLM Agents - Giskard-AI/giskard-oss

This tool automatically generates test sets to evaluate retrieval-augmented generation (RAG) applications that use large language models. It's designed for developers and data scientists building RAG systems who need… Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes. Evaluate LLM-based RAG Applications with automated test set generation launched in 2024; newer entries below may have overtaken it.

1
Giskard▲521
Open-source testing framework for LLMs & ML models
2023 · github.com · its alternatives →
2
Ragas – Open-source library for evaluating RAG pipelines▲121
Ragas is an open-source library for evaluating and testing RAG and other LLM applications. Github: https://docs.ragas.io/en/stable/, docs: https://docs.ragas.io/. Ragas provides you with different sets of metrics and methods like synthetic test data generation to help you evaluate your RAG applications. Ragas started off by scratching our own itch for evaluating our RAG chatbots last year. Problems Ragas can solve - How do you choose the best components for your RAG, such as the retriever, reranker, and LLM? - How do you formulate a test dataset…
2024 · github.com · its alternatives →
3
Turn any website into a knowledge base for LLMs▲305
I built this tool because I wanted a way to just take a bunch of URLs or domains, and query their content in RAG applications. It takes away the pain of crawling, extracting content, chunking, vectorizing, and updating periodically. I'm curious to see if it can be useful to others. I meant to launch this six months ago but life got in the way...
2024 · embedding.io · its alternatives →
4
RAGstack▲111
Deploy a private ChatGPT alternative hosted within your VPC
2023 · github.com · its alternatives →
5
Demystifying Advanced RAG Pipelines▲131
I've built an advanced RAG (Retrieval-Augmented Generation) pipeline from scratch to demystify the complex mechanics of modern LLM-powered Question Answering systems. This repository features: -- An implementation of a sub-question query engine from scratch to answer complex user questions. -- Illustrative explanations that unveil the inner workings of the system. -- An analysis of the challenges I faced while working with the system, like prompt engineering and cost estimation. -- Qualitative comparison with similar frameworks like LlamaIndex, offering a broader perspective. Key Takeaway:…
2023 · github.com · its alternatives →
6
Ragas – Open-source library for evals and testing RAG systems▲15
Ragas is an open-source library designed for evaluating and testing RAG (Retrieval-Augmented Generation) and other LLM applications. It offers a diverse set of metrics and methods, including synthetic test data generation, to help you assess your RAG applications. Ragas was initially developed to address our own needs for evaluating RAG chatbots last year. ### Problems Ragas Can Solve: - How can you select the best components for your RAG, such as the retriever, reranker, and LLM? - How can you create a test dataset without incurring significant expenses and time? We believe there's a need…
2024 · github.com · its alternatives →
7
LLMWare▲358
Dev tool to make AI apps to deploy privately or locally
2024 · llmware.ai · its alternatives →
8
PromptTools – open-source tools for evaluating LLMs and vector DBs▲211
Hey HN! We’re Kevin and Steve. We’re building PromptTools (https://github.com/hegelai/prompttools): open-source, self-hostable tools for experimenting with, testing, and evaluating LLMs, vector databases, and prompts. Evaluating prompts, LLMs, and vector databases is a painful, time-consuming but necessary part of the product engineering process. Our tools allow engineers to do this in a lot less time. By “evaluating” we mean checking the quality of a model's response for a given use case, which is a combination of testing and benchmarking. As examples: - For generated…
2023 · github.com · its alternatives →
9
Improve LLM Performance by Maximizing Iterative Development▲104
I have been working in AI space for a while now, first at FAANG with ML since 2021, then with LLM in start-ups since early 2023. I think LLM Application development is extremely iterative, more so than any other types of development. This is because to improve an LLM application performance (accuracy, hallucinations, latency, cost), you need to try various combinations of LLM models, prompt templates (e.g., few-shot, chain-of-thought), prompt context with different RAG architecture, different agent architecture, and more. There are thousands of possible combinations and you need a process…
2024 · github.com · its alternatives →
10
Giskard – Testing framework dedicated to LLMs and Tabular ML models▲15
🐢 Open-Source Evaluation & Testing library for LLM Agents - Giskard-AI/giskard-oss
2023 · github.com · its alternatives →
11
R2R – Open-source framework for production-grade RAG▲167
Hello HN, I'm Owen from SciPhi (https://www.sciphi.ai/), a startup working on simplifying˛Retrieval-Augmented Generation (RAG). Today we’re excited to share R2R (https://github.com/SciPhi-AI/R2R), an open-source framework that makes it simpler to develop and deploy production-grade RAG systems. Just a quick reminder: RAG helps Large Language Models (LLMs) use current information and specific knowledge. For example, it allows a programming assistant to use your latest documents to answer questions. The idea is to gather all the relevant information…
2024 · github.com · its alternatives →
12
How we leapfrogged traditional vector based RAG with a 'language map'▲162
TL;DR: Vector-based RAG performs poorly for many real-world applications like codebase chats, and you should consider 'language maps'. Part of our mission at Mutable.ai is to make it much easier for developers to build and understand software. One of the natural ways to do this is to create a codebase chat, that answer questions about your repo and help you build features. It might seem simple to plug in your codebase into a state-of-the-art LLM, but LLMs have two limitations that make human-level assistance with code difficult: 1. They currently have context windows that are too small to…
2024 · twitter.com · its alternatives →
13
Create LLM graders and run evals in JavaScript with one file▲28
Hi HN! Run it: OPENROUTER_API_KEY="sk" npx bff-eval --demo We built a tool to help people take LLM outputs and easily grade them / eval them to know how good an assistant response is. We've built a number of LLM apps, and while we could ship decent tech demos, we were disappointed with how they'd perform over time. We worked with a few companies who had the same problem, and found out scientifically building prompts and evals is far from a solved problem... writing these things feels more like directing a play than coding. Inspired by Anthropic's constitutional ai concepts, and amazing…
2025 · github.com · its alternatives →
14
Verta RAG System▲84
Go from dataset to custom RAG prototype in 5 minutes
2024 · cloudera.com · its alternatives →
15
Rag About It▲48
Dive deep into AI Retrieval Augmented Generation (RAG)
2024 · ragaboutit.com · its alternatives →
16
RagMetrics▲13
We built live AI evaluation to detect hallucinations, grounding issues, and drift as outputs are generated. This helps teams move GenAI from pilots into production.
2025 · ragmetrics.ai · its alternatives →
17
Tonic Validate Metrics – an open-source RAG evaluation metrics package▲40
Hey HN, Joe and Ethan from Tonic.ai here. We just released a new open-source python package for evaluating the performance of Retrieval Augmented Generation (RAG) systems. Earlier this year, we started developing a RAG-powered app to enable companies to talk to their free-text data safely. During our experimentation, however, we realized that using such a new method meant that there weren’t industry-standards for evaluation metrics to measure the accuracy of RAG performance. We built Tonic Validate Metrics (tvalmetrics, for short) to easily calculate the benchmarks we needed to meet in…
2023 · github.com · its alternatives →
18
RAG-based AI assistant in PHP▲16
An AI assistant built with PHP, Solr and LLM backend of choice. Proof of concept mostly. - Krisseck/php-rag
2023 · github.com · its alternatives →
19
AutoRAG – AutoML tool for RAG. Find an optimal RAG pipeline▲18
- Discovering the most effective RAG pipeline for your specific data and use case can be daunting. It requires experimenting with various RAG modules and configurations, which are both time-consuming and complex. - AutoRAG addresses this challenge by automatically evaluating different combinations of RAG modules and their parameters. You don't need to write implementation code yourself; everything is set up through a single YAML file. - Our aim is to save you the hassle of continuously adapting to new RAG modules and configurations. Instead, you can focus on developing robust data for your…
2024 · github.com · its alternatives →
20AF
A free AI risk assessment tool for LLM applications▲37
We’ve built an AI risk assessment tool designed specifically for GenAI/LLM applications. It's still early, but we’d love your feedback. Here’s what it does: 1. it performs comprehensive AI risk assessments by analyzing your codebase against different AI regulation/framework or even internal policies. It identifies potential issues and suggests fixes directly through one click PRs. 2. the first framework the platform supports is OWASP Top 10 for LLM Applications 2025, upcoming framework will be ISO 42001 as well as custom policy documents. 3. we're a small, early stage team, so the…
2025 · gettavo.com · its alternatives →
21
FiddleCube – Generate Q&A to test your LLM▲78
Convert your vector embeddings into a set of questions and their ideal responses. Use this dataset to test your LLM and catch failures caused by prompt or RAG updates. Get started in 3 lines of code: ``` pip3 install fiddlecube ``` ``` from fiddlecube import FiddleCube fc = FiddleCube(api_key="") dataset = fc.generate( [ "The cat did not want to be petted.", "The cat was not happy with the owner's behavior.", ], 10, ) dataset ``` Generate your API key: https://dashboard.fiddlecube.ai/api-key # Ideal QnA datasets for testing, eval and training LLMs Testing, evaluation or…
2024 · github.com · its alternatives →
22
Linq▲35
Build search engine for texts, images and beyond In 1 hour
2023 · getlinq.com · its alternatives →
23
Open-source model and scorecard for measuring hallucinations in LLMs▲65
Hi all! This morning, we released a new Apache 2.0 licensed model on HuggingFace for detecting hallucinations in retrieval augmented generation (RAG) systems. What we've found is that even when given a "simple" instruction like "summarize the following news article," every LLM that's available hallucinates to some extent, making up details that never existed in the source article -- and some of them quite a bit. As a RAG provider and proponents of ethical AI, we want to see LLMs get better at this. We've published an open source model, a blog more thoroughly describing our methodology (and…
2023 · vectara.com · its alternatives →
24
I implemented evals metrics for LLMs that runs locally on your machine▲22
The LLM Evaluation Framework. Contribute to confident-ai/deepeval development by creating an account on GitHub.
2023 · github.com · its alternatives →

Also compare

Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →