Shadow Testing for Decision/or Algorithms
Hi HN! I’m a founder at Nextmv (YC 20) [1] We’ve been building out optimization algorithm testing capabilities (acceptance tests, etc.) and just shipped our first pass at shadow testing [2, 3]. In our experience, tools like shadow testing save time and build confidence in decision models, but tools also take time to build and maintain. We’ve seen shadow testing tools in the machine learning and MLOps space [4], but not so much in the operations research community. A lot of folks here [5] seem experienced with optimization models and we’d love to have your feedback! What do you like? What…
Shadow Testing for Decision/or Algorithms launched on September 5, 2023 with 7 votes, #1,586 of 4,228 launches that month and more than 50% of that year's launches. The site was still online in September 2026, 3 years on.
What it does
Shadow testing isn’t just for the MLOps crowd. Decision algorithm teams can have it too for their optimization models solving routing, scheduling, allocation, and fulfillment problems across supply chain, retail, healthcare, and more.
We're thrilled to announce our first release of shadow testing in Nextmv! Shadow testing is a useful way to de-risk algorithm rollout and build confidence in deploying model changes to production systems. Nextmv users can now run a test model alongside a production model using the same online inputs and production conditions with the goal of analyzing and sharing model performance results. Check out this video for a quick walkthrough and read on for more details. This latest release builds on a lot of Nextmv engineering effort and customer feedback from the last few months. In our experience, it's one thing to say you want to do shadow testing (or other types of tests), it's another to…from nextmv.io
In the maker’s words, at launch
Hi HN! I’m a founder at Nextmv (YC 20) [1] We’ve been building out optimization algorithm testing capabilities (acceptance tests, etc.) and just shipped our first pass at shadow testing [2, 3]. In our experience, tools like shadow testing save time and build confidence in decision models, but tools also take time to…
Does a similar job
all alternatives →
TensorZero2025 · github.com · ▲307Hi HN! We're Gabriel & Viraj, and we're excited to open source TensorZero. To be a little cheeky, TensorZero is an open-source platform that helps LLM applications graduate from API wrappers into defensible AI products. 1. Integrate our model gateway 2. Send metrics or feedback 3. Unlock compounding improvements in quality, cost, and latency It enables a data & learning flywheel for LLMs by unifying: • Inference: one API for all LLMs, with <1ms P99 overhead • Observability: inference & feedback → your database • Optimization: better prompts, models, inference strategies • Experimentation:…
TrailsJan 2026 · trails-red.vercel.app · ▲81Hey all, Justin here. I previously built Phind, the AI search engine for developers. One of the biggest problems we had there was figuring out what went wrong with bad searches. We had tons of searches per day, but less than 1% of users gave any explicit feedback. So we were either manually digging through searches or making general system improvements and hoping they helped. This problem gets harder with agents. Traces are longer and more complex. It takes more effort to review them, so I'm building a tool that lets you analyze LLM outputs directly to help developers of LLM apps and agents…
Spec27Apr 2026 · spec27.ai · ▲34Hi HN! We’re a team of ML validation specialists and we’ve been building /Spec27, a tool for testing whether AI agents still do their job safely and reliably as models, prompts, tools, and surrounding systems change. We started working on this because a lot of current LLM evaluation work seems aimed at scoring general model behavior, while many teams are deploying systems that have a specific mission to fulfill. Many of the tools also assume you have full access to the agent stack and traces so you can place SDKs and Gateways, but a lot of agents are being created on vendor platforms…
Prompt-Engineering Tool: AI-to-AI Testing for LLM2023 · github.com · ▲37Spelltest framework simulates conversations between AI ‘synthetic users' in an environment to test and refine LLM-based applications. It ensures your app converse with utmost accuracy and relevance. Post-chat, Spelltest assesses responses, providing qualitative and quantitative feedback on performance. Suitable for both chat and completion modes. When to use: - After modifying your prompt. - When your LLM provider updates. - As a CI step for you repo. All feedback and collaborations appreciated!
Argmin AIJul 2026 · argminai.com · ▲32Hey, HN community! We've built Argmin AI after shipping LLM features where the demo worked, then the bill and latency got unpredictable in production. Prompts expanded, context grew, retrieval got noisy, retries appeared, and agent workflows added loops. Argmin AI optimizes LLM-related expenses as a system: 1. prompt and context efficiency 2. model selection and routing 3. RAG inefficiencies and caching opportunities 4. agent workflows (tool calls, retries, loop control) Changes are validated with evals and guardrails (tests, gates, judges), tailored to your quality definition and goals.…
Vibe-check skillJul 2026 · github.com · ▲11Howdy friends. I've been a zero to one product manager for 12 plus years and along the way I kept borrowing pieces from different discovery frameworks until I had a methodology of my own, one that actually raises the odds of an idea succeeding. This does not replace user interviews obviously, but it leads you in the right direction. Looking forward for your feedback
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 21d ago · stillwet.art
PixVerse R2▲381A real-time world model you can explore and change
AI · 30d ago · world.pixverse.video
Source: Show HN post ↗ · launch text quoted in part
