LLM, a Rust Crate/CLI for CPU Inference of LLMs (LLaMA, GPT-NeoX, etc.)
G'day, HN! I'm one of the maintainers of `llm`. I've been working alongside a trusty group of contributors to bring this project to life, and we're now at a point where we're ready to share it with the world. Large language models (LLMs) are taking the computing world by storm due to their emergent abilities that allow them to perform a wide variety of tasks, including translation, summarization, code generation, and even some degree of reasoning. However, the ecosystem around LLMs is still in its infancy, and it can be difficult to get started with these models. `llm` is a one-stop shop for…
In plain words
LLM is an open-source Rust crate and command-line tool for running large language models like LLaMA and GPT-NeoX on CPUs. It enables developers and users to perform inference tasks such as translation, summarization, and code generation without relying on external services. The tool can be embedded directly into applications or used standalone, making it accessible to those looking to integrate language models locally.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
G'day, HN! I'm one of the maintainers of `llm`. I've been working alongside a trusty group of contributors to bring this project to life, and we're now at a point where we're ready to share it with the world. Large language models (LLMs) are taking the computing world by storm due to their emergent abilities that allow them to perform a wide variety of tasks, including translation, summarization, code generation, and even some degree of reasoning. However, the ecosystem around LLMs is still in its infancy, and it can be difficult to get started with these models. `llm` is a one-stop shop for running inference on large language models (of the kind that power ChatGPT and more); we provide a CLI and a Rust crate for running inference on these models, all entirely open-source. The crate can be embedded in your own projects, allowing you to easily integrate LLMs into your own applications. We hope that `llm` can help to alleviate some of the pain points that users face when working with LLMs. Our goal is to build a robust solution for inferencing on LLMs that users can rely on for their projects, so that we can provide a moment of peace in the chaos of the LLM ecosystem. At present, we are powered by `ggml` (similar to `llama.cpp`), but we intend to add additional backends in the near-future. This means that we currently only support CPU inference, but we have several ideas in mind for how to add GPU support, as well as other accelerators. We're looking for feedback on the project, and we'd love to hear from you! If you're interested in contributing, please reach out to us on our Discord (https://discord.gg/YB9WaXYAWU), or post an issue on our GitHub (https://github.com/rustformers/llm/issues).
Does a similar job
all alternatives →- MLMy LLM CLI tool can run tools now, from Python code or plugins2025 · simonwillison.net · ▲529
- LTLLM-Term – Simple Rust-based CLI assist tool2024 · github.com · ▲32
Current AI-assisted CLI tools are often part of larger systems and work better on Linux. I built llm-term to address these. It's a Rust-based tool that compiles into a single binary file. You only need to download the binary, add it to your PATH, and configure your OpenAI key to get started. While llm-term offers an option for gpt-4o, it works great with gpt-4o-mini. So it's not costly. I appreciate any feedback or suggestions.
- OROllama – Run LLMs on your Mac2023 · github.com · ▲284
Hi HN A few folks and I have been working on this project for a couple weeks now. After previously working on the Docker project for a number of years (both on the container runtime and image registry side), the recent rise in open source language models made us think something similar needed to exist for large language models too. While not exactly the same as running linux containers, running LLMs shares quite a few of the same challenges. There are "base layers" (e.g. models like Llama 2), specific configuration to run correctly (parameters, temperature, context window sizes etc). There's…
- TVTiny-vLLM – high performance LLM inference engine in C++ and CUDAMay 2026 · github.com · ▲205
- AEAn educational Local Qwen3 LLM Inference project written in Rust2025 · github.com · ▲15
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 18d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 19d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 28d ago · cactuscompute.com
Launched alongside, May 2023
the whole month →
- BR
In today's world, catchy headlines and articles often distract readers from the facts and relevant information. By utilizing OpenAI's language models, Boring Report processes sensationalist news articles, transforms them into the content you see, and helps readers focus on the essential details. We recently updated our iOS app experience, so any and all feedback would be appreciated. App Link: https://apps.apple.com/us/app/boring-report-news-by-ai/id644...
AI · 2023 · boringreport.org
Momentum Page▲834Launch your website in seconds, get users in minutes
Dev tools · 2023 · page.mmntm.build