Run Llama.cpp In-Process from Java with Project Panama FFM
A week of benchmarks on a Xeon CPU box. TurboQuant ≠ 8× faster, speculative decoding ≠ universal win, ik_llama.cpp breaks parallel calls. Gemma-4-E4B-it on stock llama.cpp wins. Eleven cells, full source, MIT.
Run Llama.cpp In-Process from Java with Project Panama FFM launched on June 5, 2026 with 6 votes, #2,066 of 22,647 launches that month and more than 75% of that year's launches.
What it does
Local LLM for Spring Boot via Project Panama FFM and llama.cpp
The only in-process, tool-calling local LLM for the JVM — Spring-first, OpenAI-compatible, llama.cpp-backed via Project Panama FFM. No JNI, no daemon, no native-install dance. Requires JDK 22+. The model runs inside your application's own process. No Ollama-style sidecar to install and supervise, no HTTP round-trip, no idle resource drain. Inference is stateful and rides your app's lifecycle, Actuator health, and Micrometer metrics. Java talks to llama.cpp through the JDK 22 Foreign Function & Memory API (GA, not incubator), over a thin ~700-LOC extern-C bridge on llama.cpp's common_chat. No hand-written JNI glue, far fewer crash vectors. Consumes upstream's official prebuilt llama.cpp…from deemwar-products.github.io
Does a similar job
all alternatives →
Low-latency local LLM runner via OpenJDK Panama FFM (Java 22)Jul 2026 · github.com · ▲38I wanted to run AI from inside the JVM. I started out with the standard REST sidecar, ripped that out to use Project Panama (Foreign Function & Memory API) in the new JDK versions to interface directly with llama.cpp. I still wasn't happy with how that functioned, so I built libargus.cc to get a clean ABI to expose a structured API up in the JVM landscape. It still uses Project Panama to interface directly with llama.cpp, whisper.cpp, and ggml compute graphs. I have zero-allocation on the hot paths, memory segments for prompts and tokens are allocated once inside confined Arenas. Raw…
Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook2023 · github.com · ▲673Locally run an Instruction-Tuned Chat-Style LLM . Contribute to antimatter15/alpaca.cpp development by creating an account on GitHub.
Open-source load balancer for llama.cpp2024 · github.com · ▲151Stateful load balancer customized for llama.cpp (with a reverse proxy).
TurboPilot: Copilot clone runs code completion LLM on your CPU2023 · github.com · ▲37Hi HN, I spent my easter weekend stuck in the house with COVID and I decided to play with llama.cpp [1] and fauxpilot [2] to see if I could get LLM code assist working on pure CPU. As a proof of concept I'd say I've proven that it's possible. However there's still a lot to do. The auto complete is quite slow at the moment. PRs welcome. [1] https://github.com/ggerganov/llama.cpp [2] https://github.com/fauxpilot/fauxpilot
Speeding up LLM inference 2x times (possibly)2024 · asciinema.org · ▲419Here's a project I've been working on for the last few months. It's a new (I think) algorithm, that allows to adjust smoothly - and in real time - how many calculations you'd like to do during inference of an LLM model. It seems that it's possible to do just 20-25% of weight multiplications instead of all of them, and still get good inference results. I implemented it to run on M1/M2/M3 GPU. The mmul approximation itself can be pushed to run 2x fast before the quality of output collapses. The inference speed is just a bit faster than Llama.cpp's, because the rest of implementation…
More work this month
the category →
Endless lofi beats composed live in your browser, over pixel-art nights in Paris, Tokyo, New York, Hong Kong, Sydney, San Francisco and more. Free, no sign-up.
Work · 24d ago · loficities.com
Smooth Recorder▲257Mac screen recordings that look edited when you hit stop
Work · 18d ago · smoothrecorder.com
Work · 12d ago · readrare.com
Source: Show HN post ↗
Launched alongside, June 2026
the whole month →
Fundraisly▲1,537AI fundraising agent that finds investors and books meetings
AI · Jun 2026 · fundraisly.com
Homebrew 6.0.0▲1,481Today, I’m proud to announce Homebrew 6.0.0. The most significant changes since 5.1.0 are a new tap trust security mechanism, the new faster, smaller, default internal Homebrew JSON API, sandboxing on Linux, better defaults informed by our user survey, many brew bundle improvements, improved performance and initial support for macOS 27 (Golden Gate). Happy to discuss any questions here!
Dev tools · Jun 2026 · brew.sh
Hi HN! I was super frustrated at apps like tiktok and wechat preventing me from saving or downloading the animated gifs, so I made a little tool to turn a screen recording into a slack-sized emoji. Hope you get some value from this!
Life & fun · Jun 2026 · vorpus.github.io
Search 18 years of Hacker News and chart how any topic, tool, or person trended. Overlay terms to compare their rise and fall across 45M posts and comments.
Life & fun · Jun 2026 · hackernewstrends.com