AI · alternatives · 2026
24 alternatives to Pytest-evals – Simple LLM apps evaluation using pytest
A pytest plugin for running and analyzing LLM evaluation tests. - AlmogBaku/pytest-evals
Pytest-evals is an evaluation framework for large language model applications built on pytest. It allows developers to test LLM apps using familiar pytest syntax and conventions. The tool integrates evaluation testing… Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes.
- 1
Built a ~9M param LLM from scratch to understand how they actually work. Vanilla transformer, 60K synthetic conversations, ~130 lines of PyTorch. Trains in 5 min on a free Colab T4. The fish thinks the meaning of life is food. Fork it and swap the personality for your own character.
Apr 2026 · github.com · its alternatives →
- 2
Hey HN! We’re Kevin and Steve. We’re building PromptTools (https://github.com/hegelai/prompttools): open-source, self-hostable tools for experimenting with, testing, and evaluating LLMs, vector databases, and prompts. Evaluating prompts, LLMs, and vector databases is a painful, time-consuming but necessary part of the product engineering process. Our tools allow engineers to do this in a lot less time. By “evaluating” we mean checking the quality of a model's response for a given use case, which is a combination of testing and benchmarking. As examples: - For generated…
2023 · github.com · its alternatives →
- 3
Hi HN! We’re Adrien and Kanav. We met at our previous job, where we spent about a third of our lives combating a constant firehose of bugs. In the hope of reducing this pain for others in the future, we’re working on automating debugging. We’re currently working on a platform that ingests logs and then automatically reproduces, root causes and ultimately fixes production bugs as they happen. You can see some of our work on this here - https://news.ycombinator.com/item?id=39528087 As we were building the root-cause phase of our automated debugger, we realized that we developed…
2024 · github.com · its alternatives →
- 4
Spelltest framework simulates conversations between AI ‘synthetic users' in an environment to test and refine LLM-based applications. It ensures your app converse with utmost accuracy and relevance. Post-chat, Spelltest assesses responses, providing qualitative and quantitative feedback on performance. Suitable for both chat and completion modes. When to use: - After modifying your prompt. - When your LLM provider updates. - As a CI step for you repo. All feedback and collaborations appreciated!
2023 · github.com · its alternatives →
- 5
LLM 0.26 is out with the biggest new feature since I started the project: support for tools. You can now use the LLM CLI tool—and Python library—to grant LLMs from …
2025 · simonwillison.net · its alternatives →
- 6
- 7
TensorZero▲307Hi HN! We're Gabriel & Viraj, and we're excited to open source TensorZero. To be a little cheeky, TensorZero is an open-source platform that helps LLM applications graduate from API wrappers into defensible AI products. 1. Integrate our model gateway 2. Send metrics or feedback 3. Unlock compounding improvements in quality, cost, and latency It enables a data & learning flywheel for LLMs by unifying: • Inference: one API for all LLMs, with <1ms P99 overhead • Observability: inference & feedback → your database • Optimization: better prompts, models, inference strategies • Experimentation:…
2025 · github.com · its alternatives →
- 8
Hi HN! Run it: OPENROUTER_API_KEY="sk" npx bff-eval --demo We built a tool to help people take LLM outputs and easily grade them / eval them to know how good an assistant response is. We've built a number of LLM apps, and while we could ship decent tech demos, we were disappointed with how they'd perform over time. We worked with a few companies who had the same problem, and found out scientifically building prompts and evals is far from a solved problem... writing these things feels more like directing a play than coding. Inspired by Anthropic's constitutional ai concepts, and amazing…
2025 · github.com · its alternatives →
- 9
Hi HN! We’re been working hard on this low-code tool for rapid prompt discovery, robustness testing and LLM evaluation. We’ve just released documentation to help new users learn how to use it and what it can already do. Let us know what you think! :)
2023 · chainforge.ai · its alternatives →
- 10
The LLM Evaluation Framework. Contribute to confident-ai/deepeval development by creating an account on GitHub.
2023 · github.com · its alternatives →
- 11
The LLM Evaluation Framework. Contribute to confident-ai/deepeval development by creating an account on GitHub.
2023 · github.com · its alternatives →
- 12
LeetCode for PyTorch — 65 ML/AI interview problems from real interviews at Google, Meta, Anthropic. Jupyter notebooks, an auto-grader, and an MCP AI tutor. - Exorust/TorchLeet
2025 · github.com · its alternatives →
- 13
- 14
Recently I've been working on making LLM evaluations fast by using bayesian optimization to select a sensible subset. Bayesian optimization is used because it’s good for exploration / exploitation of expensive black box (paraphrase, LLM). I would love to hear your thoughts and suggestions on this!
2024 · github.com · its alternatives →
- 15
Hey HN! I'm Caleb, one of the contributors to Opik, a new open source framework for LLM evaluations. Over the last few months, my colleagues and I have been working on a project to solve what we see as the most painful parts of writing evals for an LLM application. For this initial release, we've focused on a few core features that we think are the most essential: - Simplifying the implementation of more complex LLM-based evaluation metrics, like Hallucination and Moderation. - Enabling step-by-step tracking, such that you can test and debug each individual component of your LLM application,…
2024 · github.com · its alternatives →
- 16
- 17
- 18
How pytest, unittest, and nose deal with assertions.
2016 · pythontesting.net · its alternatives →
- 19
DemoGPT▲94Auto generative AI app generator with the power of Llama 2
2023 · github.com · its alternatives →
- 20
Convert your vector embeddings into a set of questions and their ideal responses. Use this dataset to test your LLM and catch failures caused by prompt or RAG updates. Get started in 3 lines of code: ``` pip3 install fiddlecube ``` ``` from fiddlecube import FiddleCube fc = FiddleCube(api_key="") dataset = fc.generate( [ "The cat did not want to be petted.", "The cat was not happy with the owner's behavior.", ], 10, ) dataset ``` Generate your API key: https://dashboard.fiddlecube.ai/api-key # Ideal QnA datasets for testing, eval and training LLMs Testing, evaluation or…
2024 · github.com · its alternatives →
- 21
Hi, we are working on a tool for speeding up test runs, by skipping tests unaffected by code changes. Effectivly, Saving 80-95% of the time, by skipping 80-95% of tests. We started a few months ago, and have managed to get into a few production CI systems. All our prospects and users are on holiday right now. So we decided to repackage and open-source for local test running. available here (https://github.com/nabaz-io/nabaz) under MIT license. One line change: pytest -v -> nabaz test --cmdline "pytest -v" Stalk us on GitHub, or just Star us. Ask questions, we'll answer in…
2022 · github.com · its alternatives →
- 22
Hello HN! Pierre and Paul here. We are building an open source text analytics tool for user inputs and LLM app outputs The repo is https://github.com/phospho-app/phospho and landing is https://phospho.ai Most people building with LLMs today don’t have quantified evaluation and usage metrics on the interactions between users and their product. The only solution is to read every message (or a sample) to get a sense of what is going on. You can't improve your product without understanding who your users are and how they are using it. Nobody would launch a website…
2024 · github.com · its alternatives →
- 23
The fastest way to learn pytest.
2017 · pragprog.com · its alternatives →
- 24
A pytest-based framework for testing multi AI agents systems
2025 · github.com · its alternatives →
Also compare
- I built a tiny LLM to demystify how language models work alternatives
- PromptTools – open-source tools for evaluating LLMs and vector DBs alternatives
- Leaping – Debug Python tests instantly with an LLM debugger alternatives
- Prompt-Engineering Tool: AI-to-AI Testing for LLM alternatives
- My LLM CLI tool can run tools now, from Python code or plugins alternatives
- PyText alternatives
Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →