Alternatives
Products that do what Benchmark your team's AI coding security posture does
Hi HN. We launched a free AI Coding Risk Assessment tool to help engineering teams and businesses benchmark the security and compliance posture of their AI coding workflows and policies against peers in the industry. This anonymous 24-question survey delivers: - A 0–100 risk score that measures your AI coding security posture - A live benchmark that compares your AI-assisted development practices with peers - A research-based checklist that identifies improvement areas We're seeing more and more clients signal their concerns about the sudden increase of source code written by AI coding…
- 1
- 2
- 3
- 4
- 5CB
AI agents now have impressive reasoning capabilities. This raises an important question: how dangerous are these AI agents at identifying & exploiting web vulnerabilities? We created CVE-bench to find out (I'm one contributor of 16). To our knowledge CVE-bench is the first benchmark using real-world web vulnerabilities to evaluate AI agents' cyberattack capabilities. We included 40 CVEs from NIST's database, focusing on critical-severity vulnerability (CVSS > 9.0). To properly evaluate agents’ attacks, we built isolated environments with containerization and identified 8 common attack…
2025 · github.com
- 6MD
We’re excited to share ML-Dev-Bench, a new open-source benchmark that tests AI agents on real-world ML development tasks. Unlike typical coding challenges or Kaggle-style competitions, our benchmark simulates end-to-end ML workflows including: - Dataset handling and preprocessing - Debugging model and code failures - Implementing new model architectures - Fine-tuning and improving existing models With 30 diverse tasks, ML-Dev-Bench evaluates agents across critical stages of ML development. To complement this, we built Calipers, a framework that provides systematic performance evaluation and…
2025 · github.com
- 7IE
Hey HN, when building ML systems for industrial AI, we have learned that data inspection is critical during the ML development process. We are also big fans of the Hugging Face ecosystem. That is why we built an integration to our data exploration tool Spotlight that allows you to interactively explore Hugging Face datasets with one line of code. Spotlight lets you leverage model results such as predictions and embeddings to gain a deeper understanding in data segments and model failure modes. Currently, many many NLP, CV, Audio and multimodal datasets are supported both locally and on the…
2023 · huggingface.co
- 8IB
Hi all, I'm sure some of the best engineers out here are having a hard time standing out nowadays. It's hard to evaluate and improve your skills, when AI is writing the code. Especially when a junior dev is sitting by your side and "accomplishing" 2x more than you. I didn't like this reality where the line between real talent and AI slop is blurring, so I decided to create a challenge, purely for the community, that is made to truly give a stage for talented devs to stand out in the age of AI. We encourage devs to bring their agents with them, because the challenge is built to not be…
May 2026 · theincidentchallenge.com
- 9AM
Jul 2026 · github.com
- 10CB
I built a small benchmark to test CLI coding agents on blind bug detection. A challenger agent injects bugs and writes ground truth (`bugs.json`). A different reviewer agent audits the repo without seeing ground truth, and an LLM matcher scores bug-to-finding assignments. Current run: 50 repos, 150 challenges, 450 reviews, 2,603 injected bugs. Weighted detection: Claude 58.05%, Codex 37.84%, Gemini 27.81%. LLM-judge benchmarks are easy to get wrong, so I’d really appreciate critical feedback on benchmark fairness, scoring/matching methodology, and obvious failure modes I’m missing. Full…
Feb 2026 · github.com
- 11
We built an open sourced coordination layer for AI agents working on the same repository. Detects work duplication and design conflicts early
9d ago · twing.dev
- 12TS
Code review is the new bottleneck. "Tests passing" is no longer sufficient to trust the changes, and the (human) cost of evaluating the quality and robustness of new agent-written contributions is skyrocketing. We built Topos to evaluate code quality based on the structural properties of the programs themselves. We map your files to graphs (AST, CFG, CPG, MDG) and calculate metrics that can characterize how simple, composable, or secure your programs are. Agents can use this tool as they write and optimize based on your preferences. And yes, the inspiration for the repository is from…
Jun 2026 · krv.ai
- 13OS
Hey HN! I'm one of the cofounders of Sourcebot, an open source alternative to Sourcegraph. Sourcebot lets you index thousands of repos across multiple platforms (GitHub, GitLab, Bitbucket), and gives you a powerful interface to search across them. You can learn more in our original HN launch post: https://news.ycombinator.com/item?id=41711032 We just added an AI code review agent that reviews your PRs and automatically detects issues that a human reviewer may have missed. We've been using an AI code review agent for a few weeks now, and it regularly catches issues that we…
2025 · docs.sourcebot.dev
- 14SA
I built SpecMind, an open source developer tool for spec driven vibe coding. It keeps architecture and implementation aligned from the first commit instead of letting them drift apart. With AI assistants writing more of our code, projects move faster but architectural consistency is often lost. Each developer or AI can introduce new patterns, and after a few sprints, the structure becomes fragmented. SpecMind helps prevent that by generating and maintaining living architecture specs directly from your code. It works in three steps: 1. analyze – scans your codebase and generates…
Nov 2025 · github.com
- 15CT
Hi HN community, We're developing a free tool that aims to help founders and product teams estimate the costs of developing an MVP versus a full-scale product. Here's the gist: 1. You input a description of your idea into our AI Estimator. 2. It provides you with two estimates: one for an MVP and another for a complete product. We're sharing it here because we value the critical and insightful feedback that the HN community is known for. The tool is in its early stages, and we're looking to refine its accuracy and user experience. If you have a moment, please try it out:…
2023 · allcancode.com
- 16HP
Hi HN! I'm building Hopsule. If you use AI coding tools like Cursor, Copilot, or Claude, you’ve probably seen this happen: The AI writes good code - but it ignores your architecture. It doesn’t know: - why you chose a specific pattern - which conventions your team agreed on - which decisions are already locked in So it falls back to generic patterns, outdated examples, or random GitHub training data. Over time this slowly breaks the consistency of the codebase. Most teams try to fix this with: - giant Markdown files - wiki pages - long prompts - Slack threads But those aren't…
Mar 2026
- 17TA
Hello HN, I am Brian Cardinale, a penetration tester and security researcher at SecureCoders. We have been performing more and more AI based security assessments. We were presented a unique challenge of testing a system where the only interface was voice based, and as much as I like talking on the phone , we decided to create a test harness to facilitate the actual testing in a more systematic way. The technical test harness was the easy part, though. Creating test goals and attack strategies to help facilitate repeated and comprehensive testing became the real challenge. As such, we have…
Feb 2026 · redcaller.com
- 18BY
Hey HN - We're building wispbit (https://wispbit.com/) - a tool that lets you build your own AI code reviewer. We built this because we worked in big and complex codebases where we kept hitting booby traps - often the same ones. People forgot things, or quit altogether, amplifying the problem. We looked for other ways to fix this, but the solution is usually a combination of: - Writing a linter rule - too difficult and time consuming. - Writing docs and having frequent meetings on alignment - basically a full time job. - Using plug and play code reviewers - too generic and…
2025
- 19BA
I built CodeLens.AI - a tool that compares how 6 top LLMs (GPT-5, Claude Opus 4.1, Claude Sonnet 4.5, Grok 4, Gemini 2.5 Pro, o3) handle your actual code tasks. How it works: - Upload code + describe task (refactoring, security review, architecture, etc.) - All 6 models run in parallel (~2-5 min) - See side-by-side comparison with AI judge scores - Community votes on winners (blind voting) - Each evaluation gets reflected in the overall AI model leaderboard, showing us best ones Why I built this: Existing benchmarks (HumanEval, SWE-Bench) don't reflect real-world developer tasks. I wanted to…
Oct 2025 · codelens.ai
- 20SF
Hi HN, Over the past two years I’ve built and debugged a fair number of production pipelines—mainly retrieval‑augmented generation stacks, agent frameworks, and multi‑step reasoning services. A pattern emerged: most incidents weren’t outright crashes, but silent structural faults that slowly compromised relevance, accuracy, or stability. I began logging every recurring fault in a shared notebook. Colleagues started using the list for post‑mortems, so I turned it into a small public reference: 16 distinct failure modes (semantic drift after chunking, embedding/meaning mismatches,…
2025 · github.com
- 21TS
Hello Hacker News community! I'm currently working in financial risk management within the banking sector, and I began my career as a Data Science specialist. For quite some time, my friend and I have been developing a small pet project just for fun. This tool has repeatedly helped us save time when testing various hypotheses and machine learning models. The core idea is to combine different scripts—created in various programming languages and virtual environments—within a minimalist graphical interface. Whether you're building models, running a local neural network, or sending requests to…
2024
- 22OB
Today, we're launching the Open Benchmarks Grants: a $3M commitment to fund open-source and academic teams building benchmarks for AI agents. In partnership with HuggingFace, PrimeIntellect, FactoryHQ, Together, Harbor, and PyTorch, the grants provide funding, data development support, and research collaboration. Our ability to measure AI has been outpaced by our ability to develop it, and we believe this evaluation gap is one of the most important problems in AI. Open benchmarks are one of the most important levers for advancing AI safely and responsibly—but the academic and open-source…
Feb 2026 · benchmarks.snorkel.ai
- 23SM
I work as an engineer at the Dutch government. We have hundreds of technical standards that developers should follow when building government software: API design rules, messaging protocols, authentication profiles, accessibility requirements. The problem is that most developers don't know these standards exist until someone reviews their code (if at all). Skills are Markdown files that inject domain knowledge into AI coding tools. When a developer starts building an API, the tool automatically loads the relevant standard. No plugins to write, no code. Just structured knowledge in Markdown.…
Feb 2026 · anneschuth.nl
- 24TN
Hi guys, I’m excited to share an update on ReproModel, an open-source toolbox designed to streamline the testing and reproduction of machine learning models. I, like many of you, have really struggled with benchmarking and comparing models, from missing code, to opaque experiment parameters slowing the process. I decided to take matters into my own hands, and created a mini-toolbox in my free time to streamline the process. The goal is to reduce the time and effort spent on replicating experiments, enabling researchers to focus on innovation rather than setup. Knowing this task is not an…
2024 · github.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →