Dev tools · alternatives · 2026

24 alternatives to We made our own inference engine for Apple Silicon

We wrote our inference engine on Rust, it is faster than llama cpp in all of the use cases. Your feedback is very welcomed. Written from scratch with idea that you can add support of any kernel and platform.

This is a Rust-based inference engine designed for Apple Silicon, built from scratch to run large language models. It targets developers looking to deploy AI models on Mac hardware. The engine claims performance… Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes.

1
Open-source engine running Gemma 4 26B in 2 GB RAM on any M-series Mac▲919
Hi HN, I built a specialized inference engine for running 4-bit Gemma 4 26B-A4B-IT on any M-series Mac using about 2 GB of RAM. It is called TurboFieldfare and is written in Swift and Metal. I have always adored on-device AI. It feels like magic that you can run a powerful NN on your Mac or iPhone. So I wanted to push the limits a bit and run a model whose weights don’t fit in memory. The model’s 4-bit quantized weights occupy roughly 14 GB, which makes running it with conventional inference tools almost impossible on an 8 GB or even 16 GB Mac once the OS, applications, and KV cache are…
Jul 2026 · github.com · its alternatives →
2
Speeding up LLM inference 2x times (possibly)▲419
Here's a project I've been working on for the last few months. It's a new (I think) algorithm, that allows to adjust smoothly - and in real time - how many calculations you'd like to do during inference of an LLM model. It seems that it's possible to do just 20-25% of weight multiplications instead of all of them, and still get good inference results. I implemented it to run on M1/M2/M3 GPU. The mmul approximation itself can be pushed to run 2x fast before the quality of output collapses. The inference speed is just a bit faster than Llama.cpp's, because the rest of implementation…
2024 · asciinema.org · its alternatives →
3
Tiny-vLLM – high performance LLM inference engine in C++ and CUDA▲205
Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM - jmaczan/tiny-vllm
May 2026 · github.com · its alternatives →
4
Inference Engine by GMI Cloud▲180
Fast multimodal-native inference at scale
Dec 2025 · gmicloud.ai · its alternatives →
5
Ollama v0.19▲410
Massive local model speedup on Apple Silicon with MLX
Apr 2026 · ollama.com · its alternatives →
6
KVSplit – Run 2-3x longer contexts on Apple Silicon▲272
I discovered that in LLM inference, keys and values in the KV cache have very different quantization sensitivities. Keys need higher precision than values to maintain quality. I patched llama.cpp to enable different bit-widths for keys vs. values on Apple Silicon. The results are surprising: - K8V4 (8-bit keys, 4-bit values): 59% memory reduction with only 0.86% perplexity loss - K4V8 (4-bit keys, 8-bit values): 59% memory reduction but 6.06% perplexity loss - The configurations use the same number of bits, but K8V4 is 7× better for quality This means you can run LLMs with 2-3× longer…
2025 · github.com · its alternatives →
7IB
I built a toy TPU that can do inference and training on the XOR problem▲134
We wanted to do something very challenging to prove to ourselves that we can do anything we put our mind to. The reasoning for why we chose to build a toy TPU specifically is fairly simple: - Building a chip for ML workloads seemed cool - There was no well-documented open source repo for an ML accelerator that performed both inference and training None of us have real professional experience in hardware design, which, in a way, made the TPU even more appealing since we weren't able to estimate exactly how difficult it would be. As we worked on the initial stages of this project, we…
2025 · tinytpu.com · its alternatives →
8SA
Spice.ai OSS 1.0 – data query and AI-inference engine built in Rust▲26
2025 · spiceai.org · its alternatives →
9
An educational Local Qwen3 LLM Inference project written in Rust▲15
An educational Rust project for exporting and running inference on Qwen3 LLM family - reinterpretcat/qwen3-rs
2025 · github.com · its alternatives →
10
NobodyWho▲106
Hey, Hello, we are the team behind NobodyWho; an open-source plugin that embeds large language models functionality inside Unity and Godot games - completely locally and offline. We try to make it as easy and user friendly as possible to use our plugin, so feel free to create an issue on our GitHub if you have any inconveniences - and we will try to improve it! So why might this interest you: - Written in Rust with llama.cpp bindings and optional GPU back-ends (Vulkan / Metal) for fast inference. - Drop-in nodes / components: add a model asset and a chat or embedding node and get a…
Aug 2026 · github.com · its alternatives →
11
Gemma 3 inference in pure C++ with Metal acceleration▲6
Pure C++23 LLM inference for Apple Silicon chips. Contribute to ybubnov/metalchat development by creating an account on GitHub.
Jul 2026 · github.com · its alternatives →
12
LLM, a Rust Crate/CLI for CPU Inference of LLMs (LLaMA, GPT-NeoX, etc.)▲45
G'day, HN! I'm one of the maintainers of `llm`. I've been working alongside a trusty group of contributors to bring this project to life, and we're now at a point where we're ready to share it with the world. Large language models (LLMs) are taking the computing world by storm due to their emergent abilities that allow them to perform a wide variety of tasks, including translation, summarization, code generation, and even some degree of reasoning. However, the ecosystem around LLMs is still in its infancy, and it can be difficult to get started with these models. `llm` is a one-stop shop for…
2023 · github.com · its alternatives →
13
Inferencer – Run and deeply control local AI models (macOS release)▲15
Private inference app that lets you see the token entropy, explore and change the token probabilities. Just released on macOS, iOS version next then other platforms. Here's a demo of it in action running DeepSeek Terminus: https://youtu.be/kts098EL2PQ Would love to hear any feedback or feature requests from the community.
2025 · inferencer.com · its alternatives →
14
Composable middleware for LLM inference Optimization Passes▲7
Hey HN, I've been building AutoAgents, an AI agent framework in Rust. Today I'm sharing a feature I haven't seen done well elsewhere: composable middleware layers for LLM inference pipelines. The problem Every agent framework lets you swap LLM providers. Almost none of them give you a structured way to enforce safety, caching, or data sanitization in the inference path itself. You end up with guardrails as application-level if-statements, caching bolted on as a separate service, and PII handling as a "we'll add it later" TODO that never ships. This gets worse with local models. Cloud APIs…
Mar 2026 · github.com · its alternatives →
15
Bhumi–OSS Python Library w Rust Underhead for 2.5x Faster LLM Inference▲8
Read the full blogpost at https://rach.codes/blog/Introducing-Bhumi (click on reader to see the technical breakdown!) AI inference should be fast, but in practice it’s painfully slow. Inference bottlenecks slow down LLM-powered chatbots and AI workflows everywhere. I built Bhumi to fix that. Bhumi is a Python library designed for developers, yet its performance-critical core is implemented in Rust (via PyO3) for near-native speed. This hybrid approach delivers up to 2.5x faster response times across providers like OpenAI, Anthropic, and Gemini—without changing the…
2025 · bhumi.trilok.ai · its alternatives →
16
Lightweight Llama3 Inference Engine – CUDA C▲12
Hey, recently I took inspiration from llama.cpp, ollama, and many other similar tools that enable inference of LLMs locally, and I just finished building a Llama inference engine for the 8B model in CUDA C. I recently wanted to explore my newly founded interest in CUDA programming and my passion for machine learning. This project only makes use of the native CUDA runtime api and cuda_fp16. The inference takes place in fp16, so it requires around 17-18GB of VRAM (~16GB for model params and some more for intermediary caches). It doesn’t use cuBLAS or any similar libraries since I wanted to be…
2025 · github.com · its alternatives →
17
SwiftAI – open-source library to easily build LLM features on iOS/macOS▲74
We built SwiftAI, an open-source Swift library that lets you use Apple’s on-device LLMs when available (Apple opened access in June), and fall back to a cloud model when they aren’t available — all without duplicating code. SwiftAI gives you: - A single, model-agnostic API - An agent/tool loop - Strongly-typed structured outputs - Optional chat state Backstory: We started experimenting with Apple’s local models because they’re free (no API calls), private, and work offline. The problem: not all devices support them (older iPhones, Apple Intelligence disabled, low battery, etc.). That…
2025 · github.com · its alternatives →
18
Spice.ai Open Source 1.0-stable▲5
A portable data query and LLM-inference engine built in Rust
2025 · github.com · its alternatives →
19TF
The fastest way to run Mixtral 8x7B on Apple Silicon Macs▲18
I’d originally launched my app: Private LLM[1][2] on HN around 10 months ago, with a single RedPajama Chat 3B model. The app has come a long way since then. About a month ago, I added support for 4-bit OmniQuant quantized Mixtral 8x7B Instruct model, and it seems to outperform Q4 models at inference speed and Q8 models at text generation quality, while consuming only about 24GB of RAM[3] at 8k context length. The trick is: a) to use a better quantization algorithm and b) to use unquantized embeddings and the MoE gates (the overhead is quite small). Other notable features include many more…
2024 · its alternatives →
20
We Beat MLPerf: Modern Storage for KV Offload and LLM Training▲35
High performance storage engine for efficient LLM inference and GPU Training.
Sep 2026 · theopenlake.com · its alternatives →
21
gline-rs – an inference engine for GLiNER models, in Rust▲5
Hi everyone, wanted to share about gline-rs, an inference engine for GLiNER models written in Rust. This family of lightweight language models proved to be efficient at zero-shot Named Entity Recognition (NER) and other tasks such as Relation Extraction, while consuming less resources than large generative models (LLMs). This implementation has been written from the ground up in Rust, and supports both span- and token-oriented variants (for inference only). The goal is to provide a production-grade and user-friendly API in a modern and safe programming language, including a clean and…
2025 · github.com · its alternatives →
22
Shimmy – 5MB privacy-first, local alternative to Ollama (680MB)▲14
⚡ Pure-Rust WebGPU inference engine — OpenAI-API compatible, GGUF native, runs on any GPU. No Python. No llama.cpp. Single binary. - Michael-A-Kuykendall/shimmy
2025 · github.com · its alternatives →
23
Qwen3.6-35B-A3B on a 16 GB M1 Pro with SSD-streamed MoE▲24
Metal-first MoE inference for Apple Silicon with bounded SSD expert streaming and local OpenAI Chat, Responses, and Anthropic Messages endpoints. - hebrus-labs/hebrus
Jul 2026 · github.com · its alternatives →
24
Run 500B+ Parameter LLMs Locally on a Mac Mini▲17
Hi HN, I built OpenGraviton, an open-source AI inference engine that pushes the limits of running extremely large LLMs on consumer hardware. By combining 1.58-bit ternary quantization, dynamic sparsity with Top-K pruning and MoE routing, and mmap-based layer streaming, OpenGraviton can run models far larger than your system RAM—even on a Mac Mini. Early benchmarks: TinyLlama-1.1B drops from ~2GB (FP16) to ~0.24GB with ternary quantization. At 140B scale, models that normally require ~280GB fit within ~35GB packed. Optimized for Apple Silicon with Metal + C++ tensor unpacking, plus…
Mar 2026 · github.com · its alternatives →

Also compare

Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →