AI · alternatives · 2026
24 alternatives to We wrote a book on LLM system evals with a bear and fox
Hey all! @sridatta and I wrote a book/zine called Forest Friends on system evals for LLM-driven apps. But it's a bit more whimsical, a bit more visual, and very much inspired by the meme of LLMs being a shoggoth polished into a smiley face with RLHF. LLM system evals are important as companies move past the flashy AI demos to reliable production apps. System evals keep coming up as the answer for what you "should do", but it's not exactly a standard part of the software engineering toolkit. So we pulled from @sridatta's seven years as a research engineer at Google, plus a ton of best…
Forest Friends is a visual guide and zine on system evaluations for large language model applications. Created by researchers with experience in AI evaluation, it covers how to test and measure LLM-driven products in… Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes. We wrote a book on LLM system evals with a bear and fox launched in 2024; newer entries below may have overtaken it.
- 1
Learn to build system evals for your LLM-driven app
2024 · forestfriends.tech · its alternatives →
- 2
Hi HN! Run it: OPENROUTER_API_KEY="sk" npx bff-eval --demo We built a tool to help people take LLM outputs and easily grade them / eval them to know how good an assistant response is. We've built a number of LLM apps, and while we could ship decent tech demos, we were disappointed with how they'd perform over time. We worked with a few companies who had the same problem, and found out scientifically building prompts and evals is far from a solved problem... writing these things feels more like directing a play than coding. Inspired by Anthropic's constitutional ai concepts, and amazing…
2025 · github.com · its alternatives →
- 3
The LLM Evaluation Framework. Contribute to confident-ai/deepeval development by creating an account on GitHub.
2023 · github.com · its alternatives →
- 4
Hey HN! I'm Caleb, one of the contributors to Opik, a new open source framework for LLM evaluations. Over the last few months, my colleagues and I have been working on a project to solve what we see as the most painful parts of writing evals for an LLM application. For this initial release, we've focused on a few core features that we think are the most essential: - Simplifying the implementation of more complex LLM-based evaluation metrics, like Hallucination and Moderation. - Enabling step-by-step tracking, such that you can test and debug each individual component of your LLM application,…
2024 · github.com · its alternatives →
- 5
Hi all! This morning, we released a new Apache 2.0 licensed model on HuggingFace for detecting hallucinations in retrieval augmented generation (RAG) systems. What we've found is that even when given a "simple" instruction like "summarize the following news article," every LLM that's available hallucinates to some extent, making up details that never existed in the source article -- and some of them quite a bit. As a RAG provider and proponents of ethical AI, we want to see LLMs get better at this. We've published an open source model, a blog more thoroughly describing our methodology (and…
2023 · vectara.com · its alternatives →
- 6
I’ve been building LLM tooling for a small VC fund and found myself explaining the same mental model over and over to non-technical people around me: how a stateless LLM becomes a chatbot, how tool use works, what an agent is mechanically, and why context windows shape all of it. I never found a guide that covered that full chain at the level I wanted, so I wrote one. It’s nine short chapters, each building on the last. Deliberately simplified: the goal is a useful mental model, not a textbook. Feedback, corrections, and contributions welcome: github.com/ymyke/aiaiai
Apr 2026 · aiaiai.guide · its alternatives →
- 7
G'day, HN! I'm one of the maintainers of `llm`. I've been working alongside a trusty group of contributors to bring this project to life, and we're now at a point where we're ready to share it with the world. Large language models (LLMs) are taking the computing world by storm due to their emergent abilities that allow them to perform a wide variety of tasks, including translation, summarization, code generation, and even some degree of reasoning. However, the ecosystem around LLMs is still in its infancy, and it can be difficult to get started with these models. `llm` is a one-stop shop for…
2023 · github.com · its alternatives →
- 8IC
A friend and I wrote a book on how to build and train Deep Learning models in Go. We wanted it to be a useful reference for deep learning basics for Go programmers. Deep Learning is slowly seeping into everything we use every day and we thought it would be great if more people could do it in Go. The book is available here and on Amazon as well. https://www.packtpub.com/big-data-and-business-intelligence/hands-deep-learning-go We would appreciate any feedback and we're always looking to improve.
2019 · its alternatives →
- 9IB
I built a tool to roast landing pages with AI agents. I was gathering feedback from watching landing page roast videos, and figured out I could prompt LLMs to analyse a screenshot and roast based on the same criteria. It's not 100% accurate yet, but it has been really insightful when I've tested it on my own websites. Let me know what you think!
2024 · roastmylandingpage.io · its alternatives →
- 10IM
So hard to keep up with tooling and MLOps - I put it all in one place and got some tips from an experienced friend on what to use.
2025 · readyforagents.com · its alternatives →
- 11IB
Hey HN, I've been working on something cool that I wanted to share with you all. It's called Viewpoint, an analytics tool for LLMs like OpenAI, Anthropic models, and Gemini. The idea came from the constant flood of new LLM models and the need to figure out which ones work best for my projects without breaking the bank. With viewpoint, I can track token usage, costs, latency(WIP), and traffic over time, making it easier to compare different models and see which ones perform best and save money. The tool works asynchronously, so it doesn't add any latency to your LLM requests, and you have…
2024 · viewpointhq.com · its alternatives →
- 12
Hi all, I threw together a small prototype I am calling “Notepad.ai”. A new take on UIs for interacting with LLMs. While I enjoy using LLM’s in the chat format I wanted to see what it would be like to do it in a more long form style. It let’s you write in a pretty free form, much like Window’s Notepad, but you can choose to hit ctrl+[ to analyze the text with a preset prompt of your choosing. It has a few other small features. It’s WIP and very experimental. I would appreciate any feedback or thoughts. Video: https://youtu.be/ntdlgFmSxQY Live Demo:…
2024 · github.com · its alternatives →
- 13
The LLM Evaluation Framework. Contribute to confident-ai/deepeval development by creating an account on GitHub.
2023 · github.com · its alternatives →
- 14
Hey HN! When I started looking into LLMs and agents for software development and introducing them at work, I quickly realised that a person new to the topic faces a real barrage: - all the hype (AGI, engineers getting replaced by AI etc.) - conflicting opinions in virtually every discussion—for every person saying they’ve 10x-ed their productivity, there is a comment decrying LLMs as an utter failure - a lot of jargon (MoE, MCP, RAG, distillation, quantisation etc. etc.) - a profusion of models, IDEs/IDE extensions, CLI agents, other tools etc. Sorting through all of this can be quite…
2025 · nohypeai.dev · its alternatives →
- 15
- 16IB
I’ve been playing around with local LLMs for the past couple of months and decided to build something that can run on an iPhone. It’s a universal app built with SwiftUI and the excellent ggml library. The model is an SFT fine tuned and 4 bit quantised version of the RedPajama-INCITE-Chat-3B-v1 OSS LLM. It works reasonably well on recent-ish (~3 year old) iPhones, iPads and Macs. It was launched on the App Store yesterday[1] and Product Hunt today[2]. It seems to be reasonably ok at natural language interactions, but given its size, does pretty badly at coding and reasoning. Also, it…
2023 · its alternatives →
- 17
I’ve spent the last 2.5 months building a product that runs LLM-powered code reviews on my pull requests — and I just launched it. The tool is built specifically for solo developers. You install it on your repo, trigger a scan by creating a pull request, and it leaves structured review comments using OpenAI under the hood. Funnily enough, I used the dev version of this app to review its own pull requests while building it. It helped me spot bugs, simplify structure, and keep quality high — all with minimal need for another human in the loop. Things I want to try out in the next months : -…
2025 · codii.dev · its alternatives →
- 18
Hey HN, We’re excited to share PySpur, an open-source tool that provides a graph-based interface for building, debugging, and evaluating LLM workflows. Why we built this: Before this, we built several LLM-powered applications that collectively served thousands of users. The biggest challenge we faced was ensuring reliability: making sure the workflows were robust enough to handle edge cases and deliver consistent results. In practice, achieving this reliability meant repeatedly: 1. Breaking down complex goals into simpler steps: Composing prompts, tool calls, parsing steps, and branching…
2024 · github.com · its alternatives →
- 19LG
Hi there, I've decided to jump on the AI train and put something together with low effort & high reward, to see if it can get any traction. What do you think? Is it a promising area? Do you guys have ideas for me? There is obviously going to be sea of LLM generated content out there and one project adding up to it might not necessarily be what world needs. In the same time there is something intriguing about the area. Well, please play with it and let me know what y'all think. Much appreciated.
2023 · canonica.ai · its alternatives →
- 20
Hey folks, I’m the creator of WFGY — a semantic reasoning framework for LLMs. After open-sourcing it, I did a full technical and value audit — and realized this engine might be worth $8M–$17M based on AI module licensing norms. If embedded as part of a platform core, the valuation could exceed $30M. Too late to pull it back. So here it is — fully free, open-sourced under MIT. --- ### What does it solve? Current LLMs (even GPT-4+) lack *self-consistent reasoning*. They struggle with: - Fragmented logic across turns - No internal loopback or self-calibration - No modular thought units - Weak…
2025 · github.com · its alternatives →
- 21AA
Looking for feedback on how Props can make your life easier as an LLM application developer.
2024 · wwww.getprops.ai · its alternatives →
- 22
Why I Built This [This conversation](https://www.youtube.com/watch?v=rcz39Y0qFxw) between [Aakash Gupta](https://x.com/aakashg0) and [Jaclyn Konzelmann](https://x.com/jacalulu) (Google's Director of AI Product, on his [podcast](https://www.news.aakashg.com/s/product-growth-podcast)) truly inspired me. Jaclyn's AI PM evaluation framework was so clear and rigorous that I immediately thought: "I need to measure myself against this." "In disaster relief, you don't map safe routes and hide them. You share them." I started outlining…
Oct 2025 · aipmframework.com · its alternatives →
- 23AE
I've been working on a site [1] to give people control of their LLM workflows through AI evals - automated checks that, once defined, let you move fast without regressions and cut through hype with proof. That one-liner is aimed at software engineers, but I've spent my career helping cross-functional teams collaborate, and that's really what this is about. AI agents make powerful workflows very plausible, but only if teams can grow them incrementally without losing control - no vendor lock-in, no discipline silos, no blind trust in outputs. The site tries to meet different audiences where…
Feb 2026 · ai-evals.io · its alternatives →
- 24
Here is a production-first Keras-inspired LM framework, built with the advice of François Chollet (ex-Google, creator of Keras and ARC-AGI), our technical advisor. This system have already been deployed in production with our clients (which is why we have already every LLMOps practice implemented). It is also compatible with Jupyter and Marimo to integrate seamlessly in you Data Scientists workflows. You can try the code examples online on HF space and you can find more information in the documentation and FAQ. If you have any feedback for us don't hesitate to join our discord! More releases…
2025 · github.com · its alternatives →
Also compare
- Forest Friends Zine alternatives
- Create LLM graders and run evals in JavaScript with one file alternatives
- I implemented evals metrics for LLMs that runs locally on your machine alternatives
- Opik, an open source LLM evaluation framework alternatives
- Open-source model and scorecard for measuring hallucinations in LLMs alternatives
- Aiaiai.guide: Plain-English mental model for LLM apps, tools and agents alternatives
Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →