Low-latency local LLM runner via OpenJDK Panama FFM (Java 22)

In plain words

This Java library enables running large language models directly within the JVM using OpenJDK 22's Panama FFM API, eliminating the need for separate REST services. It interfaces with llama.cpp and whisper.cpp through a custom C library (libargus.cc) that provides a structured API while maintaining low latency through zero-allocation hot paths and pre-allocated memory arenas. The solution bundles pre-compiled native binaries for straightforward deployment, allowing Java developers to integrate AI inference into their applications with minimal overhead.

written from the facts on this page · September 2026

From the sources

A unified, unmanaged multimodal runtime. Zero-allocation native AI inference for Java 22+. - ProjectArgus-cc/libargus.cc

In the maker’s words, at launch

I wanted to run AI from inside the JVM. I started out with the standard REST sidecar, ripped that out to use Project Panama (Foreign Function & Memory API) in the new JDK versions to interface directly with llama.cpp. I still wasn't happy with how that functioned, so I built libargus.cc to get a clean ABI to expose a…

• 
  LLM Gateway2025 · llmgateway.io · ▲296

  I built an OpenAI-compatible LLM gateway that routes requests to OpenAI, Anthropic, Ollama, vLLM, llama-server, SGLang... anything that speaks /v1/chat/completions. Single Go binary, one YAML config file, no infrastructure. It does the things you'd expect from this kind of gateway... semantic routing via a three-layer cascade (keyword heuristics, embedding similarity, LLM classifier) that picks the best model when clients omit the model field, weighted round-robin load balancing across local inference servers with health checks and failover. The part I think is most…

• 
  Alpaca.cpp – Run an Instruction-Tuned Chat-Style LLM on a MacBook2023 · github.com · ▲673

  Locally run an Instruction-Tuned Chat-Style LLM . Contribute to antimatter15/alpaca.cpp development by creating an account on GitHub.

• 
  Ollama – Run LLMs on your Mac2023 · github.com · ▲284

  Hi HN A few folks and I have been working on this project for a couple weeks now. After previously working on the Docker project for a number of years (both on the container runtime and image registry side), the recent rise in open source language models made us think something similar needed to exist for large language models too. While not exactly the same as running linux containers, running LLMs shares quite a few of the same challenges. There are "base layers" (e.g. models like Llama 2), specific configuration to run correctly (parameters, temperature, context window sizes etc). There's…

• AT
  A tool to benchmark LLM APIs (OpenAI, Claude, local/self-hosted)2025 · llmapitest.com · ▲55

  I recently built a small open-source tool to benchmark different LLM API endpoints — including OpenAI, Claude, and self-hosted models (like llama.cpp). It runs a configurable number of test requests and reports two key metrics: • First-token latency (ms): How long it takes for the first token to appear • Output speed (tokens/sec): Overall output fluency Demo: https://llmapitest.com/ Code: https://github.com/qjr87/llm-api-test The goal is to provide a simple, visual, and reproducible way to evaluate performance across different LLM providers, including…

• RL
  Run Llama.cpp In-Process from Java with Project Panama FFMJun 2026 · deemwar-products.github.io · ▲6

  A week of benchmarks on a Xeon CPU box. TurboQuant ≠ 8× faster, speculative decoding ≠ universal win, ik_llama.cpp breaks parallel calls. Gemma-4-E4B-it on stock llama.cpp wins. Eleven cells, full source, MIT.

• 
  Local LLM Notepad – run a GPT-style model from a USB stick2025 · github.com · ▲40

  What it is A single 45 MB Windows .exe that embeds llama.cpp and a minimal Tk UI. Copy it (plus any .gguf model) to a flash drive, double-click on any Windows PC, and you’re chatting with an LLM—no admin rights, Cloud, or network. Why I built it Existing “local LLM” GUIs assume you can pip install, pass long CLI flags, or download GBs of extras. I wanted something my less-technical colleagues could run during a client visit by literally plugging in a USB drive. How it works PyInstaller one-file build → bundles Python runtime, llama_cpp_python, and the UI into a single PE. On first launch, it…

• 
  Spira Maxima▲578

  Video model that turns script into viral social media videos

  AI · 17d ago · spira.ai

• 
  Rill Browser▲540

  The browser where Claude Code and Codex work beside you

  AI · 17d ago · rill.love

• 
  Germany's new sovereign AI model Kolibri▲424

  Kolibri is Aleph Alpha

  AI · 19d ago · tej.as

• 
  Giving Opus 5.5 a simulated paint canvas▲392

  Paintings by AI models: programs run through a simulation of oil paint.

  AI · 20d ago · stillwet.art

• 
  PixVerse R2▲381

  A real-time world model you can explore and change

  AI · 29d ago · world.pixverse.video

• 
  Hemory▲381

  Keep listening. Searchable memory for your AI agents.

  AI · 29d ago · hemory.com

• IR
  I replaced a $120k bowling center system with $1,600 in ESP32s▲2,935

  I might be the only SRE on Earth with his own bowling center. It's a more in-depth gig than you'd think. My family and I bought an abandoned 8-lane bowling center in the rural mid-west. In our small town there weren't many recreation options for families. You've heard of a food desert? This is an R&R desert. It had been abandoned for a good reason. The roof leaks, the electrical system was constantly surging, and my 70-year-old bowling equipment (still) doesn't work perfectly. The system that keeps your score is particularly interesting to me. It's the thing you watch during your game, but…

  Life & fun · Jul 2026

• 
  Elevators▲1,680

  How elevators push your buttons

  Life & fun · Jul 2026 · john.fun

• 
  18 Words▲1,160

  Play 18 Words, the free daily word puzzle. The official home is 18words.com. Tap Play now to open today

  Life & fun · Jul 2026 · 18words.com

• BA
  Bento - An entire PowerPoint in one HTML file (edit+view+data+collab)▲1,033

  Over the past few months, our team has been building more and more slidedecks using web frontend technologies with coding harnesses like Claude Code, but a common complaint is to make even small edits we need to edit the code either manually or via the harness. To avoid this loop, I ended up creating Bento, a single HTML file with everything you need in a slide tool including animations and shared editing. There's no install or cloud login, everything works offline. The default deck is around 560 KB and it doesn't need to fetch anything once you got it. Open it in a browser and then you can…

  Dev tools · Jul 2026 · bento.page

• 
  Getting GLM 5.2 running on my slow computer▲937

  A few days ago I found myself trying out GLM 5.2 and was really positively impressed. The capabilities and security I was getting from this LLM are similar to those I've gotten from models like Claude or GPT, and this really surprised me. But then I thought, "I wonder how it would work on a normal computer like mine," and above all, "I wonder if it would work without going into OOM on a computer like mine." So I started working with the help of agents to test this possibility. I started converting the model to int4, understanding MTP usage, and if possible implementing DSA for long context.…

  AI · Jul 2026 · github.com

• 
  OpenSEO▲928

  Hello! For the last 2 months, I've been building OpenSEO. It's a user friendly UI wrapping the DataForSEO API which has high quality SEO data for keywords, backlinks, AI SEO and competitor research. Instead of paying $100+ per month, you buy DataForSEO credits and pay only for usage. The coolest thing about the project so far has been hearing about what people are building on top of it. People are using coding agents to quickly integrate with Google Search Console, Google Ads and building their own AI SEO and reporting features. Many were already building their own custom solutions around…

  Dev tools · Jul 2026 · openseo.so