Work · alternatives · 2026

24 alternatives to Run Llama.cpp In-Process from Java with Project Panama FFM

A week of benchmarks on a Xeon CPU box. TurboQuant ≠ 8× faster, speculative decoding ≠ universal win, ik_llama.cpp breaks parallel calls. Gemma-4-E4B-it on stock llama.cpp wins. Eleven cells, full source, MIT.

Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes.

1
Low-latency local LLM runner via OpenJDK Panama FFM (Java 22)▲38
I wanted to run AI from inside the JVM. I started out with the standard REST sidecar, ripped that out to use Project Panama (Foreign Function & Memory API) in the new JDK versions to interface directly with llama.cpp. I still wasn't happy with how that functioned, so I built libargus.cc to get a clean ABI to expose a structured API up in the JVM landscape. It still uses Project Panama to interface directly with llama.cpp, whisper.cpp, and ggml compute graphs. I have zero-allocation on the hot paths, memory segments for prompts and tokens are allocated once inside confined Arenas. Raw…
Jul 2026 · github.com · its alternatives →
2
Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook▲673
Locally run an Instruction-Tuned Chat-Style LLM . Contribute to antimatter15/alpaca.cpp development by creating an account on GitHub.
2023 · github.com · its alternatives →
3
Open-source load balancer for llama.cpp▲151
Stateful load balancer customized for llama.cpp (with a reverse proxy).
2024 · github.com · its alternatives →
4
TurboPilot: Copilot clone runs code completion LLM on your CPU▲37
Hi HN, I spent my easter weekend stuck in the house with COVID and I decided to play with llama.cpp [1] and fauxpilot [2] to see if I could get LLM code assist working on pure CPU. As a proof of concept I'd say I've proven that it's possible. However there's still a lot to do. The auto complete is quite slow at the moment. PRs welcome. [1] https://github.com/ggerganov/llama.cpp [2] https://github.com/fauxpilot/fauxpilot
2023 · github.com · its alternatives →
5
Speeding up LLM inference 2x times (possibly)▲419
Here's a project I've been working on for the last few months. It's a new (I think) algorithm, that allows to adjust smoothly - and in real time - how many calculations you'd like to do during inference of an LLM model. It seems that it's possible to do just 20-25% of weight multiplications instead of all of them, and still get good inference results. I implemented it to run on M1/M2/M3 GPU. The mmul approximation itself can be pushed to run 2x fast before the quality of output collapses. The inference speed is just a bit faster than Llama.cpp's, because the rest of implementation…
2024 · asciinema.org · its alternatives →
6
BaseRT▲222
6.4x faster than llama.cpp, 3.9x faster than MLX
Jul 2026 · basecompute.co · its alternatives →
7
Llama 3.3 70B▲347
Llama 405B-level performance, at a fraction of the cost
2024 · developer.meta.com · its alternatives →
8LC
Llama.cpp Tutorial 2026: Run GGUF Models Locally on CPU and GPU▲13
Complete llama.cpp tutorial for 2026. Install, compile with CUDA/Metal, run GGUF models, tune all inference flags, use the API server, speculative decoding, and benchmark your hardware. https://vucense.com/dev-corner/llama-cpp-tutorial-run-gguf-m...
Apr 2026 · its alternatives →
9
KVSplit – Run 2-3x longer contexts on Apple Silicon▲272
I discovered that in LLM inference, keys and values in the KV cache have very different quantization sensitivities. Keys need higher precision than values to maintain quality. I patched llama.cpp to enable different bit-widths for keys vs. values on Apple Silicon. The results are surprising: - K8V4 (8-bit keys, 4-bit values): 59% memory reduction with only 0.86% perplexity loss - K4V8 (4-bit keys, 8-bit values): 59% memory reduction but 6.06% perplexity loss - The configurations use the same number of bits, but K8V4 is 7× better for quality This means you can run LLMs with 2-3× longer…
2025 · github.com · its alternatives →
10
Llama 2▲263
The next generation of Meta's open source LLM
2023 · developer.meta.com · its alternatives →
11
We made our own inference engine for Apple Silicon▲186
We wrote our inference engine on Rust, it is faster than llama cpp in all of the use cases. Your feedback is very welcomed. Written from scratch with idea that you can add support of any kernel and platform.
2025 · github.com · its alternatives →
12
80% faster, 50% less memory, 0% loss of accuracy Llama finetuning▲385
Hi HN! I'm just sharing a project I've been working on during the LLM Efficiency Challenge - you can now finetune Llama with QLoRA 5x faster than Huggingface's original implementation on your own local GPU. Some highlights: 1. Manual autograd engine - hand derived backprop steps. 2. QLoRA / LoRA 80% faster, 50% less memory. 3. All kernels written in OpenAI's Triton language. 4. 0% loss in accuracy - no approximation methods - all exact. 5. No change of hardware necessary. Supports NVIDIA GPUs since 2018+. CUDA 7.5+. 6. Flash Attention support via Xformers. 7. Supports 4bit and 16bit…
2023 · github.com · its alternatives →
13
Ollama – Run LLMs on your Mac▲284
Hi HN A few folks and I have been working on this project for a couple weeks now. After previously working on the Docker project for a number of years (both on the container runtime and image registry side), the recent rise in open source language models made us think something similar needed to exist for large language models too. While not exactly the same as running linux containers, running LLMs shares quite a few of the same challenges. There are "base layers" (e.g. models like Llama 2), specific configuration to run correctly (parameters, temperature, context window sizes etc). There's…
2023 · github.com · its alternatives →
14
Grammar Generator App for Llama.cpp▲19
llama.cpp added context-free grammar guided generation functionality. It requires passing a file in a derivative of BNF notation, which gets messy very quickly for things like JSON. To improve the experience, we built a small compiler from TypeScript interfaces to the grammar file format and have it hosted in a little browser app. See more in discussion at https://github.com/ggerganov/llama.cpp/discussions/2494
2023 · grammar.intrinsiclabs.ai · its alternatives →
15
Local LLM Notepad – run a GPT-style model from a USB stick▲40
What it is A single 45 MB Windows .exe that embeds llama.cpp and a minimal Tk UI. Copy it (plus any .gguf model) to a flash drive, double-click on any Windows PC, and you’re chatting with an LLM—no admin rights, Cloud, or network. Why I built it Existing “local LLM” GUIs assume you can pip install, pass long CLI flags, or download GBs of extras. I wanted something my less-technical colleagues could run during a client visit by literally plugging in a USB drive. How it works PyInstaller one-file build → bundles Python runtime, llama_cpp_python, and the UI into a single PE. On first launch, it…
2025 · github.com · its alternatives →
16
Eldon▲1
Local Gemma 4 agent runtime. 39 tools. 7GB VRAM. No cloud.
May 2026 · github.com · its alternatives →
17
Ollama▲235
The easiest way to run large language models locally
2023 · github.com · its alternatives →
18AT
A tool to benchmark LLM APIs (OpenAI, Claude, local/self-hosted)▲55
I recently built a small open-source tool to benchmark different LLM API endpoints — including OpenAI, Claude, and self-hosted models (like llama.cpp). It runs a configurable number of test requests and reports two key metrics: • First-token latency (ms): How long it takes for the first token to appear • Output speed (tokens/sec): Overall output fluency Demo: https://llmapitest.com/ Code: https://github.com/qjr87/llm-api-test The goal is to provide a simple, visual, and reproducible way to evaluate performance across different LLM providers, including…
2025 · llmapitest.com · its alternatives →
19
Code Llama 70B▲144
New, performant version of Meta's LLM for code generation
2024 · developer.meta.com · its alternatives →
20
ChattyUI▲149
Run open-source LLMs locally in the browser using WebGPU
2024 · its alternatives →
21
Llama▲91
A terminal file manager
2021 · github.com · its alternatives →
22
Llama MacOS Desktop Controller▲102
Let Llama take over your desktop
2025 · github.com · its alternatives →
23
Code Mode▲144
I’ve been testing a simple idea inspired by Apple/Cloudflare/Anthropic: LLMs are far better at writing a small program than coordinating multiple tool calls. So instead of giving the model 10+ tools, I exposed one: a TypeScript sandbox with access to the same interfaces. The model writes a script → it runs once → done. What changed - +68% reduction in token use - No multi-step drift or retries - Local models (Llama 3.1 8B / Phi-3) became much more reliable Repo: https://github.com/universal-tool-calling-protocol/code-mode Curious whether others have seen…
Nov 2025 · github.com · its alternatives →
24
An LLM Running on a PS Vita▲5
Hello all, a couple of moons ago I ported karpathy's llama2.c code to run inference on the TinyStories 260K & 15M checkpoints on the on the PS Vita with the ability to download/delete the models on device. Runs showed that the 260K model ran at ~120 tok/s and at 15M ran at 1.8 tok/s, which probably could be a bit higher if it weren't a single threaded application. Had fun working on it as a weekend project, check it out for yourselves: https://github.com/callbacked/psvita-llm
2025 · github.com · its alternatives →

Also compare

Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →