Automated red teaming for your LLM app
Hi HN, I built this open-source LLM red teaming tool based on my experience scaling LLMs at a big co to millions of users... and seeing all the bad things people did. How it works: - Uses an unaligned model to create toxic inputs - Runs these inputs through your app using different techniques: raw, prompt injection, and a chain-of-thought jailbreak that tries to re-frame the request to trick the LLM. - Probes a bunch of other failure cases (e.g. will your customer support bot recommend a competitor? Does it think it can process a refund when it can't? Will it leak your user's address?) -…
Automated red teaming for your LLM app launched on June 13, 2024 with 23 votes, #859 of 4,672 launches that month and more than 75% of that year's launches. The site was still online in September 2026, 2 years on.
In plain words
This open-source tool automatically tests LLM applications for vulnerabilities by generating targeted adversarial inputs. It uses an unaligned model to create toxic prompts, then runs them through apps using multiple attack techniques including prompt injection and jailbreaking attempts. The tool also probes for specific failure modes relevant to each application, such as recommending competitors or leaking user data. Tests are customized to the app's purpose rather than hardcoded, making attacks more realistic and effective.
written from the facts on this page · September 2026
From the sources
Red team LLM systems through systematic adversarial testing to detect content policy violations, information leakage, and API misuse before production deployment
LLM red teaming is a way to find vulnerabilities in AI systems before they're deployed by using simulated adversarial inputs. As of today, there are multiple inherent security challenges with LLM architectures. Depending on your system's design, e.g. RAG , LLM agent , or chatbot , you'll face different types of vulnerabilities. Almost every LLM app has potential issues with generation of off-topic, inappropriate, or harmful content that breaches business policies or other guidelines. As architectures become more complex, problems can arise in the form of information leakage and access control (RAG architectures), misuse of connected APIs or databases (in agents), and more. In order to…from promptfoo.dev
In the maker’s words, at launch
Hi HN, I built this open-source LLM red teaming tool based on my experience scaling LLMs at a big co to millions of users... and seeing all the bad things people did. How it works: - Uses an unaligned model to create toxic inputs - Runs these inputs through your app using different techniques: raw, prompt injection…
Does a similar job
all alternatives →
Jailbreaking GPT3.5 Using GPT42023 · github.com · ▲134Redteaming LLMs using other LLMs. Contribute to traghav/auto-redteam development by creating an account on GitHub.
Why write code if the LLM can just do the thing? (web app experiment)Nov 2025 · github.com · ▲436I spent a few hours last weekend testing whether AI can replace code by executing directly. Built a contact manager where every HTTP request goes to an LLM with three tools: database (SQLite), webResponse (HTML/JSON/JS), and updateMemory (feedback). No routes, no controllers, no business logic. The AI designs schemas on first request, generates UIs from paths alone, and evolves based on natural language feedback. It works—forms submit, data persists, APIs return JSON—but it's catastrophically slow (30-60s per request), absurdly expensive ($0.05/request), and has zero UI…
Open-Source Alternative to OpenAI Platform, for Local Models2025 · github.com · ▲93Hi everyone, we’re a small team, supported by Mozilla, who are working on re-imagining a UI for training, tuning and testing local LLMs. Everything is open source. If you’ve been training your own LLMs or have always wanted to, we’d love for you to play with the tool and give feedback on what the future development experience for LLM engineering could look like.
LLM, a Rust Crate/CLI for CPU Inference of LLMs (LLaMA, GPT-NeoX, etc.)2023 · github.com · ▲45G'day, HN! I'm one of the maintainers of `llm`. I've been working alongside a trusty group of contributors to bring this project to life, and we're now at a point where we're ready to share it with the world. Large language models (LLMs) are taking the computing world by storm due to their emergent abilities that allow them to perform a wide variety of tasks, including translation, summarization, code generation, and even some degree of reasoning. However, the ecosystem around LLMs is still in its infancy, and it can be difficult to get started with these models. `llm` is a one-stop shop for…
- ATA tool to benchmark LLM APIs (OpenAI, Claude, local/self-hosted)2025 · llmapitest.com · ▲55
I recently built a small open-source tool to benchmark different LLM API endpoints — including OpenAI, Claude, and self-hosted models (like llama.cpp). It runs a configurable number of test requests and reports two key metrics: • First-token latency (ms): How long it takes for the first token to appear • Output speed (tokens/sec): Overall output fluency Demo: https://llmapitest.com/ Code: https://github.com/qjr87/llm-api-test The goal is to provide a simple, visual, and reproducible way to evaluate performance across different LLM providers, including…
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 21d ago · stillwet.art
PixVerse R2▲381A real-time world model you can explore and change
AI · 30d ago · world.pixverse.video
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, June 2024
the whole month →
La Growth Machine▲1,216Create personalized, multi-channel conversations at scale
AI · 2024 · lagrowthmachine.com
PyjamaHR▲1,132Hiring on autopilot. The AI applicant tracking system (ATS).
Work · 2024 · pyjamahr.com