We wrote a book on LLM system evals with a bear and fox
Hey all! @sridatta and I wrote a book/zine called Forest Friends on system evals for LLM-driven apps. But it's a bit more whimsical, a bit more visual, and very much inspired by the meme of LLMs being a shoggoth polished into a smiley face with RLHF. LLM system evals are important as companies move past the flashy AI demos to reliable production apps. System evals keep coming up as the answer for what you "should do", but it's not exactly a standard part of the software engineering toolkit. So we pulled from @sridatta's seven years as a research engineer at Google, plus a ton of best…
In plain words
Forest Friends is a visual guide and zine on system evaluations for large language model applications. Created by researchers with experience in AI evaluation, it covers how to test and measure LLM-driven products in production environments. The guide is designed for software teams building reliable AI applications who need practical evaluation methods beyond initial demos. It combines technical expertise with an accessible, illustrated format to explain system evals as a standard engineering practice.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
Hey all! @sridatta and I wrote a book/zine called Forest Friends on system evals for LLM-driven apps. But it's a bit more whimsical, a bit more visual, and very much inspired by the meme of LLMs being a shoggoth polished into a smiley face with RLHF. LLM system evals are important as companies move past the flashy AI demos to reliable production apps. System evals keep coming up as the answer for what you "should do", but it's not exactly a standard part of the software engineering toolkit. So we pulled from @sridatta's seven years as a research engineer at Google, plus a ton of best practices from around the web, and we wrote a zine that could take you from zero to eval in a way that’s fun to read. To be clear, model evals and system evals are two different things. The former compares different models, and the latter is a metric on how well you're servicing your customer queries. When you create a system eval, you're essentially defining what "good" looks like for your system. Lots of people use "vibes-based evals" (LGTM@K). It's a good place to start and will get you further than you think. But at some point, you need system evals as you get more users and more diversity in queries. To quote: Garry Tan says "Don’t rawdog your prompts! Write evals!"https://x.com/garrytan/status/1842210665550983409 Swyx says "Production AI Engineering starts with Evals"https://x.com/latentspacepod/status/1844870676202783126 How did we end up doing a zine? Coming off collaborating on the Technium Podcast, we wanted to tackle a topic where we had deep expertise while also exercising our entrepreneurial muscles. We came up with writing a zine, inspired by Julia Evans' Wizard Zine and Sailor Mercury's Bubble Zine. Originally, we were shooting for 30 pages, but ended up with 60 pages. This was also an experiment in image generation for a product. Initially, I created illustrations by hand. Midway, I decided to switch to using Midjourney to make our deadline. I needed to generate scenes with consistent characters in a specific style set in a specific architecture, and it turned out to be hard. Initially, I would be generating images 8 to 10 hours a day. Eventually, I got better at predicting what would work, and generating a suitable image dropped to 1.5 hours. Rest assured, however, all the text is human-generated and hand-edited. The issue has been well received so far. Here are some quotes from early readers: "Thanks for this resource! It provides a comprehensive introduction to LLM evaluation systems that rings true to my daily work as an AI engineer—all in less than an hour of reading and with minimal jargon. I’ll be recommending this to my team." "I was a fan of The Poignant Guide to Ruby many years ago, so it’s great to see a playfulness brought to the world of LLMs. I’m building an evals platform that makes it as easy as possible for any developer to get started with evals. This edition has been great to make sure we get the basics and terminology right." "Here's an engaging intro to evals by @sridatta and @iamwil. They've clearly put a lot of care and effort into it, where the content is well organized with plenty of illustrations throughout. Across 60 pages, they explain model vs. system evals, vibe checks and property-based tests, designing eval criteria, aligning LLM evaluators, how to measure alignment via various metrics, how to analyze evals to improve our system, and more. Now I can just direct folks to [the zine] instead of having to write it myself haha" The zine is available now. There's a preview if you want to check the vibe. https://forestfriends.tech/assets/preview.pdf I'd love to hear any feedback on the first issue, or what other topics you'd like to see tackled in later issues. If you have questions about the process of making the zine, I'd be happy to answer those also. Here's the link: https://forestfriends.tech Here's where to buy:…
Does the same job
all alternatives →- CLCreate LLM graders and run evals in JavaScript with one file2025 · github.com · ▲28
Hi HN! Run it: OPENROUTER_API_KEY="sk" npx bff-eval --demo We built a tool to help people take LLM outputs and easily grade them / eval them to know how good an assistant response is. We've built a number of LLM apps, and while we could ship decent tech demos, we were disappointed with how they'd perform over time. We worked with a few companies who had the same problem, and found out scientifically building prompts and evals is far from a solved problem... writing these things feels more like directing a play than coding. Inspired by Anthropic's constitutional ai concepts, and amazing…
- III implemented evals metrics for LLMs that runs locally on your machine2023 · github.com · ▲22
- OAOpik, an open source LLM evaluation framework2024 · github.com · ▲86
Hey HN! I'm Caleb, one of the contributors to Opik, a new open source framework for LLM evaluations. Over the last few months, my colleagues and I have been working on a project to solve what we see as the most painful parts of writing evals for an LLM application. For this initial release, we've focused on a few core features that we think are the most essential: - Simplifying the implementation of more complex LLM-based evaluation metrics, like Hallucination and Moderation. - Enabling step-by-step tracking, such that you can test and debug each individual component of your LLM application,…
- OSOpen-source model and scorecard for measuring hallucinations in LLMs2023 · vectara.com · ▲65
Hi all! This morning, we released a new Apache 2.0 licensed model on HuggingFace for detecting hallucinations in retrieval augmented generation (RAG) systems. What we've found is that even when given a "simple" instruction like "summarize the following news article," every LLM that's available hallucinates to some extent, making up details that never existed in the source article -- and some of them quite a bit. As a RAG provider and proponents of ethical AI, we want to see LLMs get better at this. We've published an open source model, a blog more thoroughly describing our methodology (and…
- AGAiaiai.guide: Plain-English mental model for LLM apps, tools and agentsApr 2026 · aiaiai.guide · ▲7
I’ve been building LLM tooling for a small VC fund and found myself explaining the same mental model over and over to non-technical people around me: how a stateless LLM becomes a chatbot, how tool use works, what an agent is mechanically, and why context windows shape all of it. I never found a guide that covered that full chain at the level I wanted, so I wrote one. It’s nine short chapters, each building on the last. Deliberately simplified: the goal is a useful mental model, not a textbook. Feedback, corrections, and contributions welcome: github.com/ymyke/aiaiai
- LALLM, a Rust Crate/CLI for CPU Inference of LLMs (LLaMA, GPT-NeoX, etc.)2023 · github.com · ▲45
G'day, HN! I'm one of the maintainers of `llm`. I've been working alongside a trusty group of contributors to bring this project to life, and we're now at a point where we're ready to share it with the world. Large language models (LLMs) are taking the computing world by storm due to their emergent abilities that allow them to perform a wide variety of tasks, including translation, summarization, code generation, and even some degree of reasoning. However, the ecosystem around LLMs is still in its infancy, and it can be difficult to get started with these models. `llm` is a one-stop shop for…
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 27d ago · cactuscompute.com
Launched alongside, October 2024
the whole month →
Softr for Notion▲925Turn Notion databases into portals & apps with no code
Dev tools · 2024 · softr.io
One inbox for all your work discussions
Work · 2024 · generalcollaboration.com