ContextCheck
Framework for testing and evaluating LLMs, RAG & chatbots.
ContextCheck launched on November 18, 2024 with 11 votes, #1,506 of 4,990 launches that month and more than 50% of that year's launches. The site was still online in September 2026, 1 year on.
What it does
An open-source framework to test LLMs, RAGs & Chatbots.It provides tools to automatically generate queries, request completions, detect regressions, perform penetration tests, and assess hallucinations, ensuring the robustness and reliability of these systems.
MIT-licensed Framework for LLMs, RAGs, Chatbots testing. Configurable via YAML and integrable into CI pipelines for automated testing. - Addepto/contextcheck
ContextCheck is a MIT-licensed framework for testing, evaluating LLMs, RAGs, and chatbots. ContextCheck is an open-source framework designed to evaluate, test, and validate large language models (LLMs), Retrieval-Augmented Generation (RAG) systems, and AI chatbots. It offers AI testing tools to automatically generate queries, request completions, detect regressions, perform penetration tests, and assess hallucinations, ensuring the robustness and reliability of these systems. ContextCheck is fully configurable via YAML and integrates seamlessly into continuous integration (CI) pipelines for efficient prompt testing and automated validation. An LLM prompt is a specific input or query…from github.com
Does a similar job
all alternatives →
Ragas – Open-source library for evaluating RAG pipelines2024 · github.com · ▲121Ragas is an open-source library for evaluating and testing RAG and other LLM applications. Github: https://docs.ragas.io/en/stable/, docs: https://docs.ragas.io/. Ragas provides you with different sets of metrics and methods like synthetic test data generation to help you evaluate your RAG applications. Ragas started off by scratching our own itch for evaluating our RAG chatbots last year. Problems Ragas can solve - How do you choose the best components for your RAG, such as the retriever, reranker, and LLM? - How do you formulate a test dataset…
Turn any website into a knowledge base for LLMs2024 · embedding.io · ▲305I built this tool because I wanted a way to just take a bunch of URLs or domains, and query their content in RAG applications. It takes away the pain of crawling, extracting content, chunking, vectorizing, and updating periodically. I'm curious to see if it can be useful to others. I meant to launch this six months ago but life got in the way...
RAGstack – private ChatGPT for enterprise VPCs, built with Llama 22023 · github.com · ▲84Hey hacker news, We’re the cofounders at Psychic.dev (http://psychic.dev) where we help companies connect LLMs to private data. With the launch of Llama 2, we think it’s finally viable to self-host an internal application that’s on-par with ChatGPT, so we did exactly that and made it an open source project. We also included a vector DB and API server so you can upload files and connect Llama 2 to your own data. The RAG in RAGstack stands for Retrieval Augmented Generation, a technique where the capabilities of a large language model (LLM) are augmented by retrieving information…
Context GatewayMar 2026 · github.com · ▲220We built an open-source proxy that sits between coding agents (Claude Code, OpenClaw, etc.) and the LLM, compressing tool outputs before they enter the context window. Demo: https://www.youtube.com/watch?v=-vFZ6MPrwjw#t=9s. Motivation: Agents are terrible at managing context. A single file read or grep can dump thousands of tokens into the window, most of it noise. This isn't just expensive — it actively degrades quality. Long-context benchmarks consistently show steep accuracy drops as context grows (OpenAI's GPT-5.4 eval goes from 97.2% at 32k to 36.6% at 1M…
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 22d ago · stillwet.art
FastRouter.ai▲364Route requests to the right LLM for cost, latency & quality
AI · 23d ago · fastrouter.ai
Source: Product Hunt launch ↗
Launched alongside, November 2024
the whole month →
Lovable▲1,477Hey HN! Previous CERN physicist turned hacker here. We've developed a way to make AI coding actually work by systematically identifying and fixing places where LLMs typically fail in full-stack development. Today we're launching as Lovable (previously gptengineer.app) since it's such a big change. The problem? AI writing code typically make small mistakes and then get stuck. Those who tried know the frustration. We fixed most of this by mapping out where LLMs fail in full-stack dev and engineering around those pitfalls with prompt chains. Thanks to this, in all comparisons I found with: v0,…
AI · 2024 · lovable.dev
Postiz▲1,188Streamline your social media with Postiz. Schedule posts, analyze performance, and manage all accounts in one place.
Dev tools · 2024 · postiz.com
I wasn't quite sure if this qualified as "Show HN" given you can't really download it and try it out. However, dang said[0]: > If it's hardware or something that's not so easy to try out over the internet, find a different way to show how it actually works—a video, for example, or a detailed post with photos. Hopefully I did that? Additionally, I've put code and a detailed guide for the netboot computer management setup on GitHub: https://github.com/kentonv/lanparty Anyway, if this shouldn't have been Show HN, I apologize! [0]…
Dev tools · 2024 · lanparty.house
Raycast Notes▲1,058Unlock a new level productivity with AI, Cloud Sync and more
Work · 2024 · raycast.com
