Alternatives
Products that do what We wrote a book on LLM system evals with a bear and fox does
Hey all! @sridatta and I wrote a book/zine called Forest Friends on system evals for LLM-driven apps. But it's a bit more whimsical, a bit more visual, and very much inspired by the meme of LLMs being a shoggoth polished into a smiley face with RLHF. LLM system evals are important as companies move past the flashy AI demos to reliable production apps. System evals keep coming up as the answer for what you "should do", but it's not exactly a standard part of the software engineering toolkit. So we pulled from @sridatta's seven years as a research engineer at Google, plus a ton of best…
- 1CL
Hi HN! Run it: OPENROUTER_API_KEY="sk" npx bff-eval --demo We built a tool to help people take LLM outputs and easily grade them / eval them to know how good an assistant response is. We've built a number of LLM apps, and while we could ship decent tech demos, we were disappointed with how they'd perform over time. We worked with a few companies who had the same problem, and found out scientifically building prompts and evals is far from a solved problem... writing these things feels more like directing a play than coding. Inspired by Anthropic's constitutional ai concepts, and amazing…
2025 · github.com
- 2II
2023 · github.com
- 3OA
Hey HN! I'm Caleb, one of the contributors to Opik, a new open source framework for LLM evaluations. Over the last few months, my colleagues and I have been working on a project to solve what we see as the most painful parts of writing evals for an LLM application. For this initial release, we've focused on a few core features that we think are the most essential: - Simplifying the implementation of more complex LLM-based evaluation metrics, like Hallucination and Moderation. - Enabling step-by-step tracking, such that you can test and debug each individual component of your LLM application,…
2024 · github.com
- 4OS
Hi all! This morning, we released a new Apache 2.0 licensed model on HuggingFace for detecting hallucinations in retrieval augmented generation (RAG) systems. What we've found is that even when given a "simple" instruction like "summarize the following news article," every LLM that's available hallucinates to some extent, making up details that never existed in the source article -- and some of them quite a bit. As a RAG provider and proponents of ethical AI, we want to see LLMs get better at this. We've published an open source model, a blog more thoroughly describing our methodology (and…
2023 · vectara.com
- 5AG
I’ve been building LLM tooling for a small VC fund and found myself explaining the same mental model over and over to non-technical people around me: how a stateless LLM becomes a chatbot, how tool use works, what an agent is mechanically, and why context windows shape all of it. I never found a guide that covered that full chain at the level I wanted, so I wrote one. It’s nine short chapters, each building on the last. Deliberately simplified: the goal is a useful mental model, not a textbook. Feedback, corrections, and contributions welcome: github.com/ymyke/aiaiai
Apr 2026 · aiaiai.guide
- 6LA
G'day, HN! I'm one of the maintainers of `llm`. I've been working alongside a trusty group of contributors to bring this project to life, and we're now at a point where we're ready to share it with the world. Large language models (LLMs) are taking the computing world by storm due to their emergent abilities that allow them to perform a wide variety of tasks, including translation, summarization, code generation, and even some degree of reasoning. However, the ecosystem around LLMs is still in its infancy, and it can be difficult to get started with these models. `llm` is a one-stop shop for…
2023 · github.com
- 7IB
I built a tool to roast landing pages with AI agents. I was gathering feedback from watching landing page roast videos, and figured out I could prompt LLMs to analyse a screenshot and roast based on the same criteria. It's not 100% accurate yet, but it has been really insightful when I've tested it on my own websites. Let me know what you think!
2024 · roastmylandingpage.io
- 8IP
To be specific, the content is generated by a GPT-2 based model. https://amzn.to/2TCc0v2 Let me know if you have any questions :-)
2020
- 9IM
So hard to keep up with tooling and MLOps - I put it all in one place and got some tips from an experienced friend on what to use.
2025 · readyforagents.com
- 10IB
Hey HN, I've been working on something cool that I wanted to share with you all. It's called Viewpoint, an analytics tool for LLMs like OpenAI, Anthropic models, and Gemini. The idea came from the constant flood of new LLM models and the need to figure out which ones work best for my projects without breaking the bank. With viewpoint, I can track token usage, costs, latency(WIP), and traffic over time, making it easier to compare different models and see which ones perform best and save money. The tool works asynchronously, so it doesn't add any latency to your LLM requests, and you have…
2024 · viewpointhq.com
- 11AD
Hi all, I threw together a small prototype I am calling “Notepad.ai”. A new take on UIs for interacting with LLMs. While I enjoy using LLM’s in the chat format I wanted to see what it would be like to do it in a more long form style. It let’s you write in a pretty free form, much like Window’s Notepad, but you can choose to hit ctrl+[ to analyze the text with a preset prompt of your choosing. It has a few other small features. It’s WIP and very experimental. I would appreciate any feedback or thoughts. Video: https://youtu.be/ntdlgFmSxQY Live Demo:…
2024 · github.com
- 12NH
Hey HN! When I started looking into LLMs and agents for software development and introducing them at work, I quickly realised that a person new to the topic faces a real barrage: - all the hype (AGI, engineers getting replaced by AI etc.) - conflicting opinions in virtually every discussion—for every person saying they’ve 10x-ed their productivity, there is a comment decrying LLMs as an utter failure - a lot of jargon (MoE, MCP, RAG, distillation, quantisation etc. etc.) - a profusion of models, IDEs/IDE extensions, CLI agents, other tools etc. Sorting through all of this can be quite…
2025 · nohypeai.dev
- 13IB
I’ve been playing around with local LLMs for the past couple of months and decided to build something that can run on an iPhone. It’s a universal app built with SwiftUI and the excellent ggml library. The model is an SFT fine tuned and 4 bit quantised version of the RedPajama-INCITE-Chat-3B-v1 OSS LLM. It works reasonably well on recent-ish (~3 year old) iPhones, iPads and Macs. It was launched on the App Store yesterday[1] and Product Hunt today[2]. It seems to be reasonably ok at natural language interactions, but given its size, does pretty badly at coding and reasoning. Also, it…
2023
- 14IB
I’ve spent the last 2.5 months building a product that runs LLM-powered code reviews on my pull requests — and I just launched it. The tool is built specifically for solo developers. You install it on your repo, trigger a scan by creating a pull request, and it leaves structured review comments using OpenAI under the hood. Funnily enough, I used the dev version of this app to review its own pull requests while building it. It helped me spot bugs, simplify structure, and keep quality high — all with minimal need for another human in the loop. Things I want to try out in the next months : -…
2025 · codii.dev
- 15GB
Hey HN, We’re excited to share PySpur, an open-source tool that provides a graph-based interface for building, debugging, and evaluating LLM workflows. Why we built this: Before this, we built several LLM-powered applications that collectively served thousands of users. The biggest challenge we faced was ensuring reliability: making sure the workflows were robust enough to handle edge cases and deliver consistent results. In practice, achieving this reliability meant repeatedly: 1. Breaking down complex goals into simpler steps: Composing prompts, tool calls, parsing steps, and branching…
2024 · github.com
- 16AV
I feel like LLMs can help me understand anything. However, after I get a summary, I can't dive in to parts that I find interesting; can't refer to original source easily and can't control context with chatbots. This is an attempt to solve for a complete knowledge consumption experience with AI . Please give me feedback!
Oct 2025 · kerns.ai
- 17LG
Hi there, I've decided to jump on the AI train and put something together with low effort & high reward, to see if it can get any traction. What do you think? Is it a promising area? Do you guys have ideas for me? There is obviously going to be sea of LLM generated content out there and one project adding up to it might not necessarily be what world needs. In the same time there is something intriguing about the area. Well, please play with it and let me know what y'all think. Much appreciated.
2023 · canonica.ai
- 18IO
Hey folks, I’m the creator of WFGY — a semantic reasoning framework for LLMs. After open-sourcing it, I did a full technical and value audit — and realized this engine might be worth $8M–$17M based on AI module licensing norms. If embedded as part of a platform core, the valuation could exceed $30M. Too late to pull it back. So here it is — fully free, open-sourced under MIT. --- ### What does it solve? Current LLMs (even GPT-4+) lack *self-consistent reasoning*. They struggle with: - Fragmented logic across turns - No internal loopback or self-calibration - No modular thought units - Weak…
2025 · github.com
- 19AA
Looking for feedback on how Props can make your life easier as an LLM application developer.
2024 · wwww.getprops.ai
- 20AE
I've been working on a site [1] to give people control of their LLM workflows through AI evals - automated checks that, once defined, let you move fast without regressions and cut through hype with proof. That one-liner is aimed at software engineers, but I've spent my career helping cross-functional teams collaborate, and that's really what this is about. AI agents make powerful workflows very plausible, but only if teams can grow them incrementally without losing control - no vendor lock-in, no discipline silos, no blind trust in outputs. The site tries to meet different audiences where…
Feb 2026 · ai-evals.io
- 21WB
Here is a production-first Keras-inspired LM framework, built with the advice of François Chollet (ex-Google, creator of Keras and ARC-AGI), our technical advisor. This system have already been deployed in production with our clients (which is why we have already every LLMOps practice implemented). It is also compatible with Jupyter and Marimo to integrate seamlessly in you Data Scientists workflows. You can try the code examples online on HF space and you can find more information in the documentation and FAQ. If you have any feedback for us don't hesitate to join our discord! More releases…
2025 · github.com
- 22HP
Hi HN. I heard you like dev tools and AI, so we wanted to share our project that we’ve been working on. We’re working on Horizon [1] - a higher level abstraction for LLMs so that developers can spend less time trying to grapple with LLMs to make them work and more time with users. This is the starting feature set which takes an auto-ML approach to identify the optimal LLM model, hyperparameters, and prompt - instead of just giving you the tooling to figure it out yourself. You can read more about it in our documentations. Our view is that as LLMs become increasingly commoditized and prompts…
2023 · gethorizon.ai
- 23IW
Hey HN, I wrote a short book about the fundamental aspects of Clojure. The target audience is programmers with at least minimal experience who want to learn how to express solutions in Clojure (and perhaps in LISP, generally). If the price is a bit too steep for you, send me an email (address in bio), and I'll provide a discount coupon. If you can't afford it all, and believe the material may help you for studying or interview preparations, I can send you a free copy.
2025 · leanpub.com
- 24AA
This repo is the result of a debate about what kind of programming language might be appropriate if humans are no longer the primary authors. Initially the thought was "LLMs can just generate binaries directly" (this was before a more famous person had the same idea). But that on reflection seems like a bad approach because languages exist to capture program semantics that are elided by translation to machine code. The next step was to wonder if an existing "machine readable" program representation can be the target for LLM code generation. It turns out yes. This project is the result of…
Mar 2026 · github.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →