Auditi
I've been building AI agents at work and the hardest part isn't the prompts or orchestration – it's answering "is this agent actually good?" in production. Tracing tells you what happened. But I wanted to know how well it happened. So I built Auditi – it captures your LLM traces and spans and automatically evaluates them with LLM-as-a-judge + human annotation workflows. Two lines to get started: auditi.init(api_key="...") auditi.instrument() # monkey-patches OpenAI/Anthropic/Gemini Every API call is captured with full span trees, token usage, and costs. No code changes to your…
Auditi launched on February 12, 2026 with 4 votes, #2,349 of 16,627 launches that month and more than 75% of that year's launches.
What it does
Tracing + evaluation in one open-source tool. LangSmith is closed-source. Langfuse is overcomplicated. Most logging tools lack built-in eval. Auditi combines all three. 2-line auto-instrumentation captures all OpenAI, Anthropic & Google API calls. 7+ LLM-as-Judge evaluators run automatically on traces. Human annotation workflows when AI judges aren't enough. Real-time cost tracking. Turn production traces into fine-tuning datasets. Self-host with docker compose up. Python SDK, FastAPI, React.
An interface developed to continuously monitor and update AI agent performance and behaviour - deduu/auditi
In the maker’s words, at launch
I've been building AI agents at work and the hardest part isn't the prompts or orchestration – it's answering "is this agent actually good?" in production. Tracing tells you what happened. But I wanted to know how well it happened. So I built Auditi – it captures your LLM traces and spans and automatically evaluates…
Does a similar job
all alternatives →
Progress AI ObservabilityAug 2026 · telerik.com · ▲168Hi HN! We’re the team behind Fiddler, and we recently built an Agent Cache feature. It works by capturing an LLM request/response pair and, once caching is enabled, serving subsequent matching calls locally so duplicate requests never hit the provider. It’s meant to reduce three pains that show up together in agent development: token cost, feedback-loop latency, and output non-determinism. We’d love to hear your thoughts.
TrailsJan 2026 · trails-red.vercel.app · ▲81Hey all, Justin here. I previously built Phind, the AI search engine for developers. One of the biggest problems we had there was figuring out what went wrong with bad searches. We had tons of searches per day, but less than 1% of users gave any explicit feedback. So we were either manually digging through searches or making general system improvements and hoping they helped. This problem gets harder with agents. Traces are longer and more complex. It takes more effort to review them, so I'm building a tool that lets you analyze LLM outputs directly to help developers of LLM apps and agents…
- ATA tool to benchmark LLM APIs (OpenAI, Claude, local/self-hosted)2025 · llmapitest.com · ▲55
I recently built a small open-source tool to benchmark different LLM API endpoints — including OpenAI, Claude, and self-hosted models (like llama.cpp). It runs a configurable number of test requests and reports two key metrics: • First-token latency (ms): How long it takes for the first token to appear • Output speed (tokens/sec): Overall output fluency Demo: https://llmapitest.com/ Code: https://github.com/qjr87/llm-api-test The goal is to provide a simple, visual, and reproducible way to evaluate performance across different LLM providers, including…
RagMetrics2025 · ragmetrics.ai · ▲13We built live AI evaluation to detect hallucinations, grounding issues, and drift as outputs are generated. This helps teams move GenAI from pilots into production.
Oodle.ai – $10 per million agent tracesJul 2026 · oodle.ai · ▲31Hi HN, we're Kiran and Vijay! Over the past two years, we have built a columnar storage engine for observability: logs, metrics, and traces. Today, it's exciting for us to show what we've built on top of that foundation: LLM Agent Observability. Given how non-deterministic agents are, storing all traces without sampling was critical for us. But these traces tend to be in the MBs, sometimes GBs - we needed to store them inexpensively. We also needed the queries and analyses to be fast. To meet both these goals, we store them in S3 in our own parquet-like file format, and query them using AWS…
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 20d ago · stillwet.art
PixVerse R2▲381A real-time world model you can explore and change
AI · 29d ago · world.pixverse.video
Source: Product Hunt launch ↗ · launch text quoted in part
Launched alongside, February 2026
the whole month →
Lovon AI Therapy▲801Feel closer and have more fun with your partner every day!
AI · Feb 2026 · lovon.app
Hello my name is Ben Ward for the past 3 years I have been remastering the financial game Wall Street Raider created by Michael Jenkins originally on DOS in 1986. It has been a rough journey but I finally see the light at the end of the tunnel. I just recently redid the website and thought maybe the full story of how this project came to be would interest you all. Thank you for reading.
Work · Feb 2026 · wallstreetraider.com