Dev tools · alternatives · 2026
24 alternatives to Python lib to run evals across providers: OpenAI, Anthropic, etc.
Library makes requests asynchronously across models, so you can spend a lot of $$ quickly if you want XD. But seriously I hope this enables folks to create and run evals (especially safety ones) a lot easier than before.
Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes. Python lib to run evals across providers: OpenAI, Anthropic, etc. launched in 2024; newer entries below may have overtaken it.
- 1
Opper AI▲229Hey HN! After the Car Wash Test post got quite a big discussion going (400+ comments, https://news.ycombinator.com/item?id=47128138), I spent the past few weeks building a tool so anyone can run these kinds of questions and get structured results. No signup and free to use. You type a question, define answer options, pick up to 50 models at a time from a pool of 200+, and they all answer independently under identical conditions. No system prompt, structured output, same setup for every model. You can also run a debate round where models see each other's reasoning and get a…
Jul 2026 · opper.ai · its alternatives →
- 2
One unified API for all AI models like Gemini, GPT-4, DALL-E
2023 · getmerlin.in · its alternatives →
- 3
Oxlo.ai▲378Scale across AI models without scaling your bill
Jun 2026 · oxcode.ai · its alternatives →
- 4
Clevrr Computer▲195Computer use but with OpenAI and Gemini models
2024 · github.com · its alternatives →
- 5WM
Try it out! https://glhf.chat/ Hey HN! We’ve been working for the past few months on a website to let you easily run (almost) any open-source LLM on autoscaling GPU clusters. It’s free for now while we figure out how to price it, but we expect to be cheaper than most GPU offerings since we can run the models multi-tenant. Unlike Together AI, Fireworks, etc, we’ll run any model that the open-source vLLM project supports: we don’t have a hardcoded list. If you want a specific model or finetune, you don’t have to ask us for it: you can just paste the Hugging Face link in and…
2024 · glhf.chat · its alternatives →
- 6
Hey HN, I am proud to show you guys that I have built an open source alternative to Azure OpenAI services. Azure OpenAI services was born out of companies needing enhanced security and access control for using different GPT models. I want to build an OSS version of Azure OpenAI services that people could self host in their own infrastructure. "How can I track LLM spend per API key?" "Can I create a development OpenAI API key with limited access for Bob?" "Can I see my LLM spend breakdown by models and endpoints?" "Can I create 100 OpenAI API keys that my students could use in a classroom…
2023 · github.com · its alternatives →
- 7
Hi HN! I built Oxyde because I was tired of duplicating my models. If you use FastAPI, you know the drill. You define Pydantic models for your API, then define separate ORM models for your database, then write converters between them. SQLModel tries to fix this but it's still SQLAlchemy underneath. Tortoise gives you a nice Django-style API but its own model system. Django ORM is great but welded to the framework. I wanted something simple: your Pydantic model IS your database model. One class, full validation on input and output, native type hints, zero duplication. The query API is…
Mar 2026 · github.com · its alternatives →
- 8
Run GPT-OSS, Qwen, Llama, and DeepSeek through one OpenAI-compatible API with streaming and clear usage-based pricing.
May 2026 · adola.app · its alternatives →
- 9
Hi HN! Run it: OPENROUTER_API_KEY="sk" npx bff-eval --demo We built a tool to help people take LLM outputs and easily grade them / eval them to know how good an assistant response is. We've built a number of LLM apps, and while we could ship decent tech demos, we were disappointed with how they'd perform over time. We worked with a few companies who had the same problem, and found out scientifically building prompts and evals is far from a solved problem... writing these things feels more like directing a play than coding. Inspired by Anthropic's constitutional ai concepts, and amazing…
2025 · github.com · its alternatives →
- 10
We open-sourced catsu, a Python client for embedding APIs. The problem: every embedding provider has a different SDK with different bugs. OpenAI has undocumented token limits. VoyageAI's retry logic was broken until September. Cohere breaks downstream libraries every release. LiteLLM's embedding support is minimal. catsu provides: - One API for 11 providers (OpenAI, Voyage, Cohere, Jina, Mistral, Gemini, etc.) - Bundled database of 50+ models with pricing, dimensions, and benchmark scores - Built-in retry with exponential backoff - Automatic cost tracking per request - Full async support…
Dec 2025 · catsu.dev · its alternatives →
- 11
Hey HN! We've run our privacy-focused open-source inference company for a while now, and we're launching a flat monthly subscription similar to Anthropic's. It should work with Cline, Roo, KiloCode, Aider, etc — any OpenAI-compatible API client should do. The rate limits at every tier are higher than the Claude rate limits, so even if you prefer using Claude it can be a helpful backup for when you're rate limited, for a pretty low price. Let me know if you have any feedback!
2025 · synthetic.new · its alternatives →
- 12
RubyLLM provides a unified interface to OpenAI, Anthropic, Google, and DeepSeek models with a clean, elegant API that feels like Ruby. What makes it different? Beautiful interfaces: chat = RubyLLM.chat embedding = RubyLLM.embed("Ruby is elegant") image = RubyLLM.paint("a sunset over mountains") Works with multiple providers through one API: # Start with GPT chat = RubyLLM.chat(model: 'gpt-4o-mini') # Switch to Claude? No problem chat.with_model('claude-3-5-sonnet') Streaming that makes sense: chat.ask "Write a story" do |chunk| print chunk.content # Same chunk format for all providers end…
2025 · rubyllm.com · its alternatives →
- 13AB
Hello HN, new user here, so please let me know if I break some rules. Currently I've been working on training reinforcement learning agents, and OpenAI gym, while is great, runs only one agent at a time. Hence I decided to extend it. I built a wrapper around OpenAI gym, such that it now runs several environments concurrently. All while (mostly) having the same call signature as OpenAI gym. And it is published to PyPI for anyone interested. For more details, please visit: https://github.com/Chimpan-Z/agymc Feedback really appreciated! Have a good day everyone!
2020 · its alternatives →
- 14PL
Hi! I’ve been working on this automatic scanner for ML models to detect issues like underperforming data slices, overconfidence in predictions, robustness problems, and others. It supports all main Python ML frameworks (sklearn, torch, xgboost, …) and integrates with the quality assurance solution we are building at Giskard AI (https://giskard.ai) to systematically test models before putting them in production. It is still a beta and I would love to hear your feedback if you have the time to try it out. We have quite a few tutorials in the docs with ready-made colab notebooks to…
2023 · docs.giskard.ai · its alternatives →
- 15AC
Built an AI code reviewer using Letta (Python) that I can call natively from Rust applications. The interesting part: real-time streaming works perfectly across the language boundary with zero hassle using RunAgent. The agent runs in Python with persistent memory, leverages the best in house agentic memory management with Letta (Pythonic AI agent framework), and my rust code just uses it (kinda) natively, though Letta has no Rust bindings. And, streaming works like magic. No FFI, no complex bridges - just native async/streaming that feels like calling any Rust librar, but without…
2025 · medium.com · its alternatives →
- 16
Hi there, looking for feedback on my new project "Featherless.AI" The idea is to allow users to run all the models on hugging face instantly. Via the OpenAI API compatible endpoint. Why? Because its a real chore to download models and spin up GPUs, especially if you want to test multiple models. Not to mention GPUs cost multiple dollars an hour to rent. And if we want more people to use open source AI, we got to make it easier for them to try and play with all of them. So what if instead of spinning up dedicated GPUs per model (which is what every provider is doing) We can startup a LLM…
2024 · featherless.ai · its alternatives →
- 17
Hey HN, I built browser-use, an open-source alternative to OpenAI’s Operator for browser-use systems, and here’s why I think it’s better: Flexibility: You can use any LLM with our tool – Gemini, Anthropic, Qwen, Llama, DeepSeek, and more. As new models improve, so does your agent. Open Source: No need to pay $200/month or endure long waitlists – it’s free and accessible to everyone today. Custom Automation: Our Python package allows you to build actual web automations. Your LLM can gain new tools, like file uploads. Cost: Our system is 30x cheaper than Operator, e.g., when used with…
2025 · github.com · its alternatives →
- 18
Hi HN, We needed a simple way to connect to the top AI models to experiment, prototype and evaluate them. Main features: - Connect to top LLMs in few lines of code (currenly OpenAI, Anthropic and AI21 are supported) - Response meta includes tokens processed, cost and latency standardized across the models - Multi-model support: Get completitions from different models at the same time - LLM benchmark: Eevaluate models on quality, speed and cost The benchmark uses predefine questions to test AI reasoning abilities across a range of "hard" queries. The outputs are then automatically evaulauted…
2023 · github.com · its alternatives →
- 19
Hi, I'm working on a project that regroups all best AI (AIaaS) from different providers (GCP, AWS, Azure, DeepL, etc.) in one API (https://github.com/edenai/edenai-apis). I've got asked the question : why aren't you regrouping Open Source models (instead of proprietary APIs) into one repo? Well because it doesn't make sens to deploy and maintain large pytorch (or other framework) AI models (especially for document parsing, image and video moderation or speech recognition) in every solution that wants AI capabilities. So using APIs makes way more sens. Deployed OpenSource…
2023 · github.com · its alternatives →
- 20UI
Hey everyone! I am excited to share updates on four of my & my teams' open-source projects that take large-scale search systems to the next level: USearch, UForm, UCall, and StringZilla. These projects are designed to work seamlessly together, end-to-end—covering everything from indexing and AI to storage and networking. And yeah, they're optimized for x86 AVX2/512 and Arm NEON/SVE hardware. USearch [1]: Think of it as Meta FAISS on steroids. It's now quicker, supports clustering of any granularity, and offers multi-index lookups. Plus, it's got more native bindings than probably…
2023 · usearch-images.com · its alternatives →
- 21
- 22
I have built many AI agents, and all frameworks felt so bloated, slow, and unpredictable. Therefore, I hacked together a minimal library that works with JSON/dict/kwargs definitions for each step, allowing you a simpler way to define reproducible agents. It supports concurrency for up to 1000 calls/min, giving you speed and predictability in your workflows. Install pip install flashlearn Input is a list of dictionaries Simply take user inputs, API responses, and calculations from other tools and feed them to FlashLearn. user_inputs = [{"query": "When was python launched?"}]…
2025 · github.com · its alternatives →
- 23
Lovelaice▲10Multi-model AI testing, evaluation, optimization made simple
2025 · lovelaice.com · its alternatives →
- 24
Hi HN, I built OpenGraviton, an open-source AI inference engine designed to push the limits of running extremely large models on consumer hardware. The system combines several techniques to drastically reduce memory and compute requirements: • 1.58-bit ternary quantization ({-1, 0, +1}) for ~10x compression • dynamic sparsity with Top-K pruning and MoE routing • mmap-based layer streaming to load weights directly from NVMe SSDs • speculative decoding to improve generation throughput These allow models far larger than system RAM to run locally. In early benchmarks, OpenGraviton reduced…
Mar 2026 · opengraviton.github.io · its alternatives →
Also compare
Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →