Alternatives
Products that do what LLMTrace does
Know which commit blew up your LLM bill
- 1
- 2
Trace LLM requests + costs with OpenTelemetry monitoring
Oct 2025
- 3AM
I built this out of curiosity about what Claude Code was actually sending to the API. Turns out, watching your tokens tick up in real-time is oddly satisfying. Sherlock sits between your LLM tools and the API, showing you every request with a live dashboard, and auto-saved copies of every prompt as markdown and json.
Jan 2026 · github.com
- 4OO
Hey HN, Nir, Gal and Tomer here. We’re open-sourcing a set of extensions we’ve built on top of OpenTelemetry that provide visibility into LLM applications - whether it be prompts, vector DBs and more. Here’s the repo: https://github.com/traceloop/openllmetry. There’s already a decent number of tools for LLM observability, some open-source and some not. But what we found was missing for all of them is that they were closed-protocol by design, vendor-locking you to use their observability platform or their proprietary framework for running your LLMs. It’s still early in the…
2023 · github.com
- 5AT
I recently built a small open-source tool to benchmark different LLM API endpoints — including OpenAI, Claude, and self-hosted models (like llama.cpp). It runs a configurable number of test requests and reports two key metrics: • First-token latency (ms): How long it takes for the first token to appear • Output speed (tokens/sec): Overall output fluency Demo: https://llmapitest.com/ Code: https://github.com/qjr87/llm-api-test The goal is to provide a simple, visual, and reproducible way to evaluate performance across different LLM providers, including…
2025 · llmapitest.com
- 6TS
I work as a SAP Integration consultant and built this as a side project. Friction point: Most self hosted LLM observability tools require Postgres, Redis and non trivial infrastructure. Teams just want to see what their agents are actually doing in Production, that set up cost discorages adoption. Torrix runs as a single docker contained backed by SQLite. The full install is: curl -o docker-compose.yml https://raw.githubusercontent.com/torrix-ai/install/main/doc... docker compose up No external dependencies. All data stays in a local SQLite file on your machine.…
May 2026 · github.com
- 7LP
Hello hacker news, I’m the maintainer of liteLLM() - package to simplify input/output to OpenAI, Azure, Cohere, Anthropic, Hugging face API Endpoints: https://github.com/BerriAI/litellm/ We’re open sourcing our implementation of liteLLM proxy: https://github.com/BerriAI/litellm/blob/main/cookbook/proxy-... TLDR: It has one API endpoint /chat/completions and standardizes input/output for 50+ LLM models + handles logging, error tracking, caching, streaming What can liteLLM proxy do? - It’s a central place to…
2023 · github.com
- 8
- 9VS
Hey HN! We’re Emma and Chris, founders of Velvet (https://www.usevelvet.com). Velvet proxies OpenAI calls and stores the requests and responses in your PostgreSQL database. That way, you can analyze logs with SQL (instead of a clunky UI). You can also set headers to add caching and metadata (for analysis). Backstory: We started by building some more general AI data tools (like a text-to-SQL editor). We were frustrated by the lack of basic LLM infrastructure, so ended up pivoting to focus on the tooling we wanted. So many existing apps, like Helicone, were hard to use as power…
2024 · usevelvet.com
- 10
- 11YD
If you've built any web-based app in the last 15 years, you probably used something like Datadog, New Relic, Sentry, etc. to monitor and trace your app, right? Why should it be different when the app you're building happens to be using LLMs? So today we're open-sourcing OpenLLMetry-JS. It's an open protocol and SDK, based on OpenTelemetry, that provides traces and metrics for LLM JS/TS applications and can be connected to any of the 15+ tools that already support OpenTelemetry. Here's the repo: https://github.com/traceloop/openllmetry-js A few months ago we launched…
2024 · github.com
- 12
- 13
Aggregate uptime monitoring across OpenAI, Claude, and more
Apr 2026 · tools.lamatic.ai
- 14LH
I work on inference scheduling — KV cache-aware routing, load balancing across GPU workers, that kind of thing. I wanted something like k9s but for my inference stack. Nothing existed, so I built it. llmtop is a real-time terminal dashboard for LLM inference workers. It scrapes the Prometheus /metrics endpoints that vLLM, SGLang, and LMCache already expose and shows everything in one view: KV cache usage, queue depth, TTFT/ITL latencies (P50/P99 from histogram buckets), token throughput, prefix cache hit rates. Color-coded — red means go fix it. ``` brew install…
Mar 2026 · github.com
- 15
- 16
- 17MY
LLM observability is an absolute must-have for anyone running something in prod (or prod-like). While all the observability startups are great, you're essentially sending all your OpenAI usage history - prompts, generations, chats - to a random third party. So this script deploys a basic proxy in your Azure account, catches all incoming OpenAI requests, stores logs in your own resource group, and comes with visualizations premade (charts, timelines, chat history, cost estimation, etc). Thanks for any thoughts and feedback!
2023 · github.com
- 18
- 19AC
Multi-tier exact-match cache for AI agents backed by Valkey or Redis. LLM responses, tool results, and session state behind one connection. Framework adapters for LangChain, LangGraph, and Vercel AI SDK. OpenTelemetry and Prometheus built in. No modules required - works on vanilla Valkey 7+ and Redis 6.2+. Shipped v0.1.0 yesterday, v0.2.0 today with cluster mode. Streaming support coming next. Existing options locked you into one tier (LangChain = LLM only, LangGraph = state only) or one framework. This solves both. npm:…
Apr 2026
- 20TO
I built TraceAIO, an open-source tool that prompts LLMs on your behalf and tells you whether ChatGPT, Perplexity, and Gemini mention your brand — and which competitors and sources show up instead. Yeah, this category smells a bit like a grift, same as early SEO. And I think over time it will become just SEO again, and become about good content. The tool just helps you monitor over time. It queries the browser products through real browser sessions, not APIs, runs on Docker, with an MCP server so you can query your own data through an LLM. No business model, Apache 2.0, self hosted. If you…
Jun 2026 · traceaio.org
- 21OS
Hi HN, Hugh and Vince here. LLMonitor helps you record, trace & search your LLM queries and chatbot conversations. You can also capture user feedback on your frontend and correlate it with backend LLM queries then use that to fine-tune your own models. The project started has an internal tool in our previous (failed) AI startup. We’re aware the LLM observability space is very crowded. Apart from being open-source, we differentiate with: - Model-agnostic and minimal lock-in (no MITM of requests). - High focus on DX and dashboard clarity. - Support for complex scenarios: e.g. a chatbot that…
2023 · github.com
- 22AU
Hi HN, I was once given the advice: Don't waste expensive frontier model credits (GPT/Claude/etc.) on bulk work. Send the boring, repetitive, high-volume jobs to a smaller model, and save the expensive prompts for when you actually need frontier-level reasoning. I complained and told my manager that I shouldnt have to think about using certain models for certain coding tasks, and that one model should handle everything. Well, here we are anyway. If anyone needs a place to absolutely abuse an LLM with high-volume tasks, come beat ours up at https://yolo-auto.com. Here are…
Jul 2026 · yolo-auto.com
- 23LR
Hi hacker news, My name is Dillion and I'm the creator of llm.report. A few months ago, I was frustrated by the lack of observability into the OpenAI API. All of us are left in the dark about API performance, latency, cost calculation, cost breakdown, and more. I just wanted to know more about how my AI app is performing in production and make data-driven decisions to improve the product. So I ended up just building it myself. There are three parts to the platform: 1. OpenAI API Dashboard (no-code) - Enter your OpenAI key and get access to detailed insights straight from the OpenAI API…
2023 · github.com
- 24SY
Hey HN, If you tried running open-source models like Llama 3.1 70B or 405B, you might have noticed that it gets very expensive. The reason looks obvious enough that you might have stopped even before trying it! - GPUs are very expensive to buy or rent - Running the most performing LLMs need 4, 8 or even 16 top of the line Nvidia GPUs - And that won’t get you anywhere near the level of VRAM needed to batch enough to get a decent throughput and efficiency Some have even questioned if open-source LLM providers are not doing some shenanigans to provide the prices they offer. VC funded…
2024
Ranked by how close each launch is in meaning, then by votes. Refine with a description →