Forest Friends Zine
Learn to build system evals for your LLM-driven app
Forest Friends Zine launched on September 24, 2024 with 9 votes, #1,401 of 5,281 launches that month and more than 50% of that year's launches. The site was still online in September 2026, 2 years on.
What it does
How do you know your improvements don't have regressions? How do you know you're driving better outcomes for your customers? This issue is a guide for AI Engineers building system evals for their LLM-driven apps.
A guide for AI Engineers building the wild world of LLM system evals
Drive improvements in your LLM-powered systems by designing, implementing, and interpreting evaluations. As engineers and product managers, learn the how to build your first eval and convince your team of the value of evals. Set in Brightwood Forest, a Large Language Model Shoggoth made its home in the canopy. Forest creatures have been using this alien intelligence to answer their questions, tell stories, and even write love letters. But integrating LLMs into complex applications is not easy: Sometimes the LLM misinterprets their instructions, struggles to understand the data it reads, or chooses to do the wrong thing entirely. Sometimes it feels like "check the vibes, cross your fingers,…from forestfriends.tech
Does a similar job
all alternatives →- WWWe wrote a book on LLM system evals with a bear and fox2024 · ▲11
Hey all! @sridatta and I wrote a book/zine called Forest Friends on system evals for LLM-driven apps. But it's a bit more whimsical, a bit more visual, and very much inspired by the meme of LLMs being a shoggoth polished into a smiley face with RLHF. LLM system evals are important as companies move past the flashy AI demos to reliable production apps. System evals keep coming up as the answer for what you "should do", but it's not exactly a standard part of the software engineering toolkit. So we pulled from @sridatta's seven years as a research engineer at Google, plus a ton of best…
Improve LLM Performance by Maximizing Iterative Development2024 · github.com · ▲104I have been working in AI space for a while now, first at FAANG with ML since 2021, then with LLM in start-ups since early 2023. I think LLM Application development is extremely iterative, more so than any other types of development. This is because to improve an LLM application performance (accuracy, hallucinations, latency, cost), you need to try various combinations of LLM models, prompt templates (e.g., few-shot, chain-of-thought), prompt context with different RAG architecture, different agent architecture, and more. There are thousands of possible combinations and you need a process…
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 22d ago · stillwet.art
FastRouter.ai▲364Route requests to the right LLM for cost, latency & quality
AI · 23d ago · fastrouter.ai
Source: Product Hunt launch ↗
Launched alongside, September 2024
the whole month →
BeforeSunset AI 2.0▲1,267Personalized AI daily planning that suits your life
AI · 2024 · beforesunset.ai
