LettuceDetect – Lightweight hallucination detector for RAG pipelines
Hallucinations are still a major blocker for deploying reliable retrieval-augmented generation (RAG) systems, especially in complex domains like medical or legal. Most existing hallucination detectors rely on full LLM inference (expensive, slow), or struggle with long-context inputs. I built LettuceDetect — an open-source, encoder-only framework that detects hallucinated spans in LLM-generated answers based on the retrieved context. No LLMs needed, and it much more efficiently. Highlights: - Token-level hallucination detection (unsupported spans flagged based on retrieved evidence) - Built…
In plain words
LettuceDetect is an open-source hallucination detector for retrieval-augmented generation systems. It identifies unsupported spans in LLM-generated answers by comparing them against retrieved context using an encoder-only framework, without requiring LLM inference. Built on ModernBERT to handle long contexts up to 4K tokens, it achieves 79.22% F1 on benchmark tests. The tool is designed for high-stakes domains like medical and legal where reliable outputs are critical.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
Hallucinations are still a major blocker for deploying reliable retrieval-augmented generation (RAG) systems, especially in complex domains like medical or legal. Most existing hallucination detectors rely on full LLM inference (expensive, slow), or struggle with long-context inputs. I built LettuceDetect — an open-source, encoder-only framework that detects hallucinated spans in LLM-generated answers based on the retrieved context. No LLMs needed, and it much more efficiently. Highlights: - Token-level hallucination detection (unsupported spans flagged based on retrieved evidence) - Built on ModernBERT — handles up to 4K token contexts - 79.22% F1 on the RAGTruth benchmark (beats previous encoder models, competitive with LLMs) - MIT licensed — Includes Python packages, pretrained models, and Hugging Face demo GitHub: https://github.com/KRLabsOrg/LettuceDetect Blog: https://huggingface.co/blog/adaamko/lettucedetect Preprint: https://arxiv.org/abs/2502.17125 Models/Demo: https://huggingface.co/KRLabsOrg Would love feedback from anyone working on RAG, hallucination detection, or efficient LLM evaluation. Also exploring real-time hallucination detection (vs. just post-gen) — open to thoughts/collab there.
Does the same job
all alternatives →- OSOpen-source model and scorecard for measuring hallucinations in LLMs2023 · vectara.com · ▲65
Hi all! This morning, we released a new Apache 2.0 licensed model on HuggingFace for detecting hallucinations in retrieval augmented generation (RAG) systems. What we've found is that even when given a "simple" instruction like "summarize the following news article," every LLM that's available hallucinates to some extent, making up details that never existed in the source article -- and some of them quite a bit. As a RAG provider and proponents of ethical AI, we want to see LLMs get better at this. We've published an open source model, a blog more thoroughly describing our methodology (and…
- DADemystifying Advanced RAG Pipelines2023 · github.com · ▲131
I've built an advanced RAG (Retrieval-Augmented Generation) pipeline from scratch to demystify the complex mechanics of modern LLM-powered Question Answering systems. This repository features: -- An implementation of a sub-question query engine from scratch to answer complex user questions. -- Illustrative explanations that unveil the inner workings of the system. -- An analysis of the challenges I faced while working with the system, like prompt engineering and cost estimation. -- Qualitative comparison with similar frameworks like LlamaIndex, offering a broader perspective. Key Takeaway:…
- TATurn any website into a knowledge base for LLMs2024 · embedding.io · ▲305
I built this tool because I wanted a way to just take a bunch of URLs or domains, and query their content in RAG applications. It takes away the pain of crawling, extracting content, chunking, vectorizing, and updating periodically. I'm curious to see if it can be useful to others. I meant to launch this six months ago but life got in the way...
- CAChainForge, a visual tool for prompt engineering and LLM evaluation2023 · chainforge.ai · ▲177
Hi HN! We’re been working hard on this low-code tool for rapid prompt discovery, robustness testing and LLM evaluation. We’ve just released documentation to help new users learn how to use it and what it can already do. Let us know what you think! :)
- FAFactual AI Q&A – Answers based on Huberman Lab transcripts2022 · huberman.rile.yt · ▲120
This is a quick prototype I built for semantic search and factual question answering using embeddings and GPT-3. It tries to solve the LLM hallucination issue by guiding it only to answer questions from the given context instead of making things up. If you ask something not covered in an episode, it should say that it doesn't know rather than providing a plausible, but potentially incorrect response. It uses Whisper to transcribe, text-embedding-ada-002 to embed, Pinecone.io to search, and text-davinci-003 to generate the answer. More examples and explanations here:…
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 27d ago · cactuscompute.com
Launched alongside, April 2025
the whole month →- IB
Hi everyone, I built PyXL — a hardware processor that executes a custom assembly generated from Python programs, without using a traditional interpreter or virtual machine. It compiles Python -> CPython Bytecode -> Instruction set designed for direct hardware execution. I’m sharing an early benchmark: a GPIO test where PyXL achieves a 480ns round-trip toggle — compared to 14-25 micro seconds on a MicroPython Pyboard - even though PyXL runs at a lower clock (100MHz vs. 168MHz). The design is stack-based, fully pipelined, and preserves Python's dynamic typing without static type restrictions.…
Dev tools · 2025 · runpyxl.com
- UC
Life & fun · 2025 · filiph.github.io
- IB
https://the-pocket.github.io/Tutorial-Codebase-Knowledge/
AI · 2025 · github.com
