AI · alternatives · 2026

24 alternatives to Bhumi–OSS Python Library w Rust Underhead for 2.5x Faster LLM Inference

Read the full blogpost at https://rach.codes/blog/Introducing-Bhumi (click on reader to see the technical breakdown!) AI inference should be fast, but in practice it’s painfully slow. Inference bottlenecks slow down LLM-powered chatbots and AI workflows everywhere. I built Bhumi to fix that. Bhumi is a Python library designed for developers, yet its performance-critical core is implemented in Rust (via PyO3) for near-native speed. This hybrid approach delivers up to 2.5x faster response times across providers like OpenAI, Anthropic, and Gemini—without changing the…

Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes.

1
We made our own inference engine for Apple Silicon▲186
We wrote our inference engine on Rust, it is faster than llama cpp in all of the use cases. Your feedback is very welcomed. Written from scratch with idea that you can add support of any kernel and platform.
2025 · github.com · its alternatives →
2
Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac▲919
Hi HN, I built a specialized inference engine for running 4-bit Gemma 4 26B-A4B-IT on any M-series Mac using about 2 GB of RAM. It is called TurboFieldfare and is written in Swift and Metal. I have always adored on-device AI. It feels like magic that you can run a powerful NN on your Mac or iPhone. So I wanted to push the limits a bit and run a model whose weights don’t fit in memory. The model’s 4-bit quantized weights occupy roughly 14 GB, which makes running it with conventional inference tools almost impossible on an 8 GB or even 16 GB Mac once the OS, applications, and KV cache are…
Jul 2026 · github.com · its alternatives →
3
Speeding up LLM inference 2x times (possibly)▲419
Here's a project I've been working on for the last few months. It's a new (I think) algorithm, that allows to adjust smoothly - and in real time - how many calculations you'd like to do during inference of an LLM model. It seems that it's possible to do just 20-25% of weight multiplications instead of all of them, and still get good inference results. I implemented it to run on M1/M2/M3 GPU. The mmul approximation itself can be pushed to run 2x fast before the quality of output collapses. The inference speed is just a bit faster than Llama.cpp's, because the rest of implementation…
2024 · asciinema.org · its alternatives →
4
Faster FastAPI with simdjson and io_uring on Linux 5.19▲290
A few months ago, I benchmarked FastAPI on an i9 MacBook Pro. I couldn't believe my eyes. A primary REST endpoint to `sum` two integers took 6 milliseconds to evaluate. It is okay if you are targeting a server in another city, but it should be less when your client and server apps are running on the same machine. FastAPI would have bottleneck-ed the inference of our lightweight UForm neural networks recently trending on HN under the title "Beating OpenAI CLIP with 100x less data and compute". (Thank you all for the kind words!) So I wrote another library. It has been a while since I have…
2023 · github.com · its alternatives →
5
Llama 3.2 Interpretability with Sparse Autoencoders▲579
I spent a lot of time and money on this rather big side project of mine that attempts to replicate the mechanistic interpretability research on proprietary LLMs that was quite popular this year and produced great research papers by Anthropic [1], OpenAI [2] and Deepmind [3]. I am quite proud of this project and since I consider myself the target audience for HackerNews did I think that maybe some of you would appreciate this open research replication as well. Happy to answer any questions or face any feedback. Cheers [1]…
2024 · github.com · its alternatives →
6
Timber – Ollama for classical ML models, 336x faster than Python▲207
Ollama for classical ML models. AOT compiler that turns XGBoost, LightGBM, scikit-learn, CatBoost & ONNX models into native C99 inference code. One command to load, one command to serve. 336x faster than Python inference. - kossisoroyce/timber
Mar 2026 · github.com · its alternatives →
7
General Compute▲315
AI models that run on an inference cloud optimized for speed
May 2026 · generalcompute.com · its alternatives →
8
TensorZero▲307
Hi HN! We're Gabriel & Viraj, and we're excited to open source TensorZero. To be a little cheeky, TensorZero is an open-source platform that helps LLM applications graduate from API wrappers into defensible AI products. 1. Integrate our model gateway 2. Send metrics or feedback 3. Unlock compounding improvements in quality, cost, and latency It enables a data & learning flywheel for LLMs by unifying: • Inference: one API for all LLMs, with <1ms P99 overhead • Observability: inference & feedback → your database • Optimization: better prompts, models, inference strategies • Experimentation:…
2025 · github.com · its alternatives →
9
80% faster, 50% less memory, 0% loss of accuracy Llama finetuning▲385
Hi HN! I'm just sharing a project I've been working on during the LLM Efficiency Challenge - you can now finetune Llama with QLoRA 5x faster than Huggingface's original implementation on your own local GPU. Some highlights: 1. Manual autograd engine - hand derived backprop steps. 2. QLoRA / LoRA 80% faster, 50% less memory. 3. All kernels written in OpenAI's Triton language. 4. 0% loss in accuracy - no approximation methods - all exact. 5. No change of hardware necessary. Supports NVIDIA GPUs since 2018+. CUDA 7.5+. 6. Flash Attention support via Xformers. 7. Supports 4bit and 16bit…
2023 · github.com · its alternatives →
10
Inference Engine by GMI Cloud▲180
Fast multimodal-native inference at scale
Dec 2025 · gmicloud.ai · its alternatives →
11
PlayHT-Turbo▲148
The fastest generative AI Text-to-Speech API
2023 · its alternatives →
12
Tiny-vLLM – high performance LLM inference engine in C++ and CUDA▲205
Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM - jmaczan/tiny-vllm
May 2026 · github.com · its alternatives →
13
LLM, a Rust Crate/CLI for CPU Inference of LLMs (LLaMA, GPT-NeoX, etc.)▲45
G'day, HN! I'm one of the maintainers of `llm`. I've been working alongside a trusty group of contributors to bring this project to life, and we're now at a point where we're ready to share it with the world. Large language models (LLMs) are taking the computing world by storm due to their emergent abilities that allow them to perform a wide variety of tasks, including translation, summarization, code generation, and even some degree of reasoning. However, the ecosystem around LLMs is still in its infancy, and it can be difficult to get started with these models. `llm` is a one-stop shop for…
2023 · github.com · its alternatives →
14
SelfHostLLM▲134
A simple calculator that estimates how many concurrent requests your GPU can handle for a given LLM, with shareable results.
2025 · selfhostllm.org · its alternatives →
15AT
A tool to benchmark LLM APIs (OpenAI, Claude, local/self-hosted)▲55
I recently built a small open-source tool to benchmark different LLM API endpoints — including OpenAI, Claude, and self-hosted models (like llama.cpp). It runs a configurable number of test requests and reports two key metrics: • First-token latency (ms): How long it takes for the first token to appear • Output speed (tokens/sec): Overall output fluency Demo: https://llmapitest.com/ Code: https://github.com/qjr87/llm-api-test The goal is to provide a simple, visual, and reproducible way to evaluate performance across different LLM providers, including…
2025 · llmapitest.com · its alternatives →
16
Reame – a CPU inference server that gets faster as it runs▲59
CPU-first LLM inference server on llama.cpp. Runs useful models on free-tier ARM boxes; rewriting the input made it ~6x faster and more accurate than tuning the engine. MIT, benchmarks and failures included. - swellweb/reame
Jul 2026 · github.com · its alternatives →
17
An educational Local Qwen3 LLM Inference project written in Rust▲15
An educational Rust project for exporting and running inference on Qwen3 LLM family - reinterpretcat/qwen3-rs
2025 · github.com · its alternatives →
18
LLM-Term – Simple Rust-based CLI assist tool▲32
Current AI-assisted CLI tools are often part of larger systems and work better on Linux. I built llm-term to address these. It's a Rust-based tool that compiles into a single binary file. You only need to download the binary, add it to your PATH, and configure your OpenAI key to get started. While llm-term offers an option for gpt-4o, it works great with gpt-4o-mini. So it's not costly. I appreciate any feedback or suggestions.
2024 · github.com · its alternatives →
19
Composable middleware for LLM inference Optimization Passes▲7
Hey HN, I've been building AutoAgents, an AI agent framework in Rust. Today I'm sharing a feature I haven't seen done well elsewhere: composable middleware layers for LLM inference pipelines. The problem Every agent framework lets you swap LLM providers. Almost none of them give you a structured way to enforce safety, caching, or data sanitization in the inference path itself. You end up with guardrails as application-level if-statements, caching bolted on as a separate service, and PII handling as a "we'll add it later" TODO that never ships. This gets worse with local models. Cloud APIs…
Mar 2026 · github.com · its alternatives →
20
Hicap▲14
One API for every model. Faster, cheaper inference.
Feb 2026 · hicap.ai · its alternatives →
21
Tilth – I spent tokens so my agents would stop wasting them (~4k Rust)▲10
I'm an "ideas person" who messes around with AI on a low budget. I got tired of watching my tokens vanish and context windows filling up while agents fumbled around trying to find the right thing. Agents don't flail like they used to with shell tools, but there are still weak/blind spots and back-and-forth episodes — especially when using tools in combination/sequence. So I built "tilth" today. Or rather, AI built it — every line is Opus 4.6. I spent a lot of my precious tokens getting it to "not shit" (at least several of the different vendors' AI overlords assure me it's not…
Feb 2026 · github.com · its alternatives →
22SA
Spice.ai OSS 1.0 – data query and AI-inference engine built in Rust▲26
2025 · spiceai.org · its alternatives →
23
AnyLLM▲20
10+ LLMs at 10x speed. Think we're kidding. Test it now!
2025 · its alternatives →
24PA
Pruna AI – Inference Optimization Engine▲6
Hello Hacker News! I am Bertrand from Pruna AI. With my associates, John, Rayan, and Stephan, we are fellow researchers in AI efficiency and reliability coming from TUM. We are building an optimization engine that combines compression methods (e.g. quantization, pruning, compilation, batching…) in the aim of saving compute power when running AI models. This optimization engine take one base model as input and returns a compressed model as output. It aims to help for two things: - Make various AI models faster and/or smaller for various hardware (because they can require significant…
2024 · its alternatives →

Also compare

Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →