AI · alternatives · 2026
24 alternatives to How we use LLMs to find testing gaps, vulnerabilities in codebases
Hello everyone! I’m thrilled to announce the latest feature from Mutahunter.ai, the ultimate tool for finding and fixing weaknesses in your code. We’ve designed Mutahunter to leverage mutation testing powered by advanced LLMs, helping you uncover vulnerabilities and enhance your code quality effortlessly. Introducing our newest feature: Detailed Mutation Testing Reports! After running our mutation tests, Mutahunter now generates comprehensive reports that clearly summarize: • Vulnerable code gaps • Test case gaps These reports significantly reduce the cognitive load on developers by…
Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes. How we use LLMs to find testing gaps, vulnerabilities in codebases launched in 2024; newer entries below may have overtaken it.
- 1
Background: We were inspired by how aider.chat was making use of pageranked AST and realized that we can use this to power high quality mutation testing using LLMs. Check it out!
2024 · github.com · its alternatives →
- 2MF
2024 · mutahunter.ai · its alternatives →
- 3
I spent a few hours last weekend testing whether AI can replace code by executing directly. Built a contact manager where every HTTP request goes to an LLM with three tools: database (SQLite), webResponse (HTML/JSON/JS), and updateMemory (feedback). No routes, no controllers, no business logic. The AI designs schemas on first request, generates UIs from paths alone, and evolves based on natural language feedback. It works—forms submit, data persists, APIs return JSON—but it's catastrophically slow (30-60s per request), absurdly expensive ($0.05/request), and has zero UI…
Nov 2025 · github.com · its alternatives →
- 4EM
Hey HN, Steven here from CodeIntegrity https://github.com/codeintegrity-ai/mutahunter We’re obsessed with automating software testing, specifically mutation testing, and have been frustrated with its slow adoption despite its proven success. I have a pretty different perspective on mutation testing and have shared my thoughts on its current state - https://www.jungsteven.com/blog/2024-07-03-past-present-future-mutation-testing Over the past few months, we’ve developed a new mutation testing tool that’s easy to use and compatible with any programming…
2024 · its alternatives →
- 5
Trails▲81Hey all, Justin here. I previously built Phind, the AI search engine for developers. One of the biggest problems we had there was figuring out what went wrong with bad searches. We had tons of searches per day, but less than 1% of users gave any explicit feedback. So we were either manually digging through searches or making general system improvements and hoping they helped. This problem gets harder with agents. Traces are longer and more complex. It takes more effort to review them, so I'm building a tool that lets you analyze LLM outputs directly to help developers of LLM apps and agents…
Jan 2026 · trails-red.vercel.app · its alternatives →
- 6
Hi all! This morning, we released a new Apache 2.0 licensed model on HuggingFace for detecting hallucinations in retrieval augmented generation (RAG) systems. What we've found is that even when given a "simple" instruction like "summarize the following news article," every LLM that's available hallucinates to some extent, making up details that never existed in the source article -- and some of them quite a bit. As a RAG provider and proponents of ethical AI, we want to see LLMs get better at this. We've published an open source model, a blog more thoroughly describing our methodology (and…
2023 · vectara.com · its alternatives →
- 7
New ways of breaking app-integrated LLMs . Contribute to greshake/llm-security development by creating an account on GitHub.
2023 · github.com · its alternatives →
- 8
Hi HN, I built this open-source LLM red teaming tool based on my experience scaling LLMs at a big co to millions of users... and seeing all the bad things people did. How it works: - Uses an unaligned model to create toxic inputs - Runs these inputs through your app using different techniques: raw, prompt injection, and a chain-of-thought jailbreak that tries to re-frame the request to trick the LLM. - Probes a bunch of other failure cases (e.g. will your customer support bot recommend a competitor? Does it think it can process a refund when it can't? Will it leak your user's address?) -…
2024 · promptfoo.dev · its alternatives →
- 9IS
Hey HN! For that last 8 months I've been trying to make agents that can hack web applications to find vulnerabilities in them - An AI Security Tester. The system has 29 agents in total, a custom LLM Orchestration framework which works on the task-subtask architecture (old-school but works amazingly for my use case, and is pretty reliable) with custom agent calling mechanism. No Auo-Gen, Langchain and Crew AI - Everything custom built for pentesting. Each test runs in an isolated Kali linux environment (on AWS Fargate), where the agents have full access to the environment to undertake any…
2025 · its alternatives →
- 10
Hey everyone, My friend and I built a simple bug fixing app that listens for alerts/issues from Sentry, contextualizes it against your codebase, and any other data sources you wish to connect (right now we support Notion, Google Docs, and Slack), and deploys an ai agent to write a PR for review in Github or Gitlab to solve the bug. Our current demo shows the end-to-end process for a trivial bug fix, but we have been testing it with open source python repos like http-pie, comparing how our agent solves a bug compared to a human engineer and it gets fairly close. We are working on adding…
2023 · resolvd.ai · its alternatives →
- 11
At testup.io we have been working for a while to bring artificial intelligence to the field of test automation. Just a few years ago, the primary challenge laid in accurately identifying UI elements following minor structural changes, such as updates to IDs or paths. The emergence of Large Language Models (LLMs) raised the bar for what it meant to be smart. Now, we anticipate the robot to do lots of things autonomously, such as retry in cases of unresponsiveness or handle minor error reports. A more challenging, but soon expected feature, would involve the test robot navigating your web shop…
2024 · github.com · its alternatives →
- 12
Spelltest framework simulates conversations between AI ‘synthetic users' in an environment to test and refine LLM-based applications. It ensures your app converse with utmost accuracy and relevance. Post-chat, Spelltest assesses responses, providing qualitative and quantitative feedback on performance. Suitable for both chat and completion modes. When to use: - After modifying your prompt. - When your LLM provider updates. - As a CI step for you repo. All feedback and collaborations appreciated!
2023 · github.com · its alternatives →
- 13
- 14CA
Imagine you have to make a little change in all your microservices, like changing a line in Dockerfile, but you have 50 microservices, ouch right! The PRs, pipelines, reviews... I have created a dashboard with a chatbot where you can simply ask "Update all repositories with alpine base image version X to Y", and then click next a bunch of time and see a list of PRs with their pipeline status. Hi friends I'm Daniel, the creator of Infra as AI. I was laid off from my SRE role a couple of months ago, and decided to build something that could always help me at work. My job was to keep the lights…
2025 · infrastructureas.ai · its alternatives →
- 15
Hey HN! I built Experiment to solve a common frustration in LLM development: the lack of proper tools for prompt engineering experimentation. Here's what makes it different: Key Features: - Load and edit chat completion logs from CSV files - Fork and modify specific conversation entries - Run inference via Anthropic, Mistral, and OpenAI - Define custom tools using JSONSchema format - Visual tool usage analysis with collapsible, sorted key-value pairs - Full mobile support and available as installable PWA Technical Highlights: - Built with React using custom isomorphic architecture -…
2025 · github.com · its alternatives →
- 16WH
Hey guys, I'm investing a ton of time on writing unit tests, for both enterprise and personal projects. I came up with the idea to make extension for AI-generated tests and cases within a VS Code. Happy to hear feedback, both positive and negative.
2024 · marketplace.visualstudio.com · its alternatives →
- 17
I’ve spent the last 2.5 months building a product that runs LLM-powered code reviews on my pull requests — and I just launched it. The tool is built specifically for solo developers. You install it on your repo, trigger a scan by creating a pull request, and it leaves structured review comments using OpenAI under the hood. Funnily enough, I used the dev version of this app to review its own pull requests while building it. It helped me spot bugs, simplify structure, and keep quality high — all with minimal need for another human in the loop. Things I want to try out in the next months : -…
2025 · codii.dev · its alternatives →
- 18
If you're interested in exploring what LLM-based agent systems these days actually do to solve certain benchmarks such as SWEBench or WebArena, we created a small leaderboard with our team, that allows to view a lot of public and OSS agent results including all the runtime traces (the step-by-step reasoning behind the scenes). Looking at traces is actually quite interesting, as they reveal a lot about the inner working and shortcomings of current agent system, e.g. see https://explorer.invariantlabs.ai/u/invariant/webarena--SteP... for an example trace.
2024 · explorer.invariantlabs.ai · its alternatives →
- 19
RagMetrics▲13We built live AI evaluation to detect hallucinations, grounding issues, and drift as outputs are generated. This helps teams move GenAI from pilots into production.
2025 · ragmetrics.ai · its alternatives →
- 20
Hi HN, I've been working with LLMs in production for a while both as a solo dev building apps for clients and working at an AI startup. The one thing that always was a pain was to pay OpenAI/Gemini/Anthropic a few dollars a month just for me to say "test" or have a CI runner validate some UI code. So I built this server called ChunkBack, that mocks the popular llm provider's functionality but allows you to type in a deterministic language: `SAY "cheese"` or `TOOLCALL "tool_name" {} "tool response"` I've had to work in some test environments and give good results for experimenting…
Nov 2025 · github.com · its alternatives →
- 21
Synthetic data generation is an essential step in training and evaluating LLMs/Agents/RAG pipelines, but tooling around this is still lacking. We're introducing Curator, an open-source library designed to streamline the data curation process. While there are many libraries to prompt LLMs, the semantics of generating synthetic data is different from prompting. For example, we need to process a large number of prompts (sometimes in millions or more) while accepting some failures, utilize several stages of prompting, incorporate human feedback, and filter out bad data using verifiers…
2025 · github.com · its alternatives →
- 22
PwnScan▲8For at least a few years (since before COVID), in my circle everbody wants an AI tool that finds vulnerabilities in binaries. I have my reservations about such a tool because of my perception that AI is overhyped, so I decided to build one myself to see how useful it could be (or not). I was actually kinda surprised how well it worked for my collection of binaries with known CVEs. It definitely has false positives and negatives, as any static analysis tool would, but I think its actually somewhat usable. I dont think the results are super actionable, but interesting nonetheless. I still…
2025 · pwnscan.com · its alternatives →
- 23
Ultra-fast, low latency LLM prompt injection/jailbreak detection ⛓️ - arekusandr/last_layer
2024 · github.com · its alternatives →
- 24
Hi guys, I’m excited to share an update on ReproModel, an open-source toolbox designed to streamline the testing and reproduction of machine learning models. I, like many of you, have really struggled with benchmarking and comparing models, from missing code, to opaque experiment parameters slowing the process. I decided to take matters into my own hands, and created a mini-toolbox in my free time to streamline the process. The goal is to reduce the time and effort spent on replicating experiments, enabling researchers to focus on innovation rather than setup. Knowing this task is not an…
2024 · github.com · its alternatives →
Also compare
- Mutahunter – LLMs to support mutating testing for all major languages alternatives
- Mutahunter – Fast, language agnostic, software testing using LLM agents alternatives
- Why write code if the LLM can just do the thing? (web app experiment) alternatives
- Extreme Mutation Testing to Optimize LLM Agent-Based Mutation Testing alternatives
- Trails alternatives
- Open-source model and scorecard for measuring hallucinations in LLMs alternatives
Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →