LLM, a Rust Crate/CLI for CPU Inference of LLMs (LLaMA, GPT-NeoX, etc.)
G'day, HN! I'm one of the maintainers of `llm`. I've been working alongside a trusty group of contributors to bring this project to life, and we're now at a point where we're ready to share it with the world. Large language models (LLMs) are taking the computing world by storm due to their emergent abilities that allow them to perform a wide variety of tasks, including translation, summarization, code generation, and even some degree of reasoning. However, the ecosystem around LLMs is still in its infancy, and it can be difficult to get started with these models. `llm` is a one-stop shop for…
LLM, a Rust Crate/CLI for CPU Inference of LLMs (LLaMA, GPT-NeoX, etc.) launched on May 9, 2023 with 45 votes, #1,507 of 4,910 launches that month and more than 90% of that year's launches. The site was still online in September 2026, 3 years on.
In plain words
LLM is an open-source Rust crate and command-line tool for running large language models like LLaMA and GPT-NeoX on CPUs. It enables developers and users to perform inference tasks such as translation, summarization, and code generation without relying on external services. The tool can be embedded directly into applications or used standalone, making it accessible to those looking to integrate language models locally.
written from the facts on this page · September 2026
From the sources
[Unmaintained, see README] An ecosystem of Rust libraries for working with large language models - rustformers/llm
In the maker’s words, at launch
G'day, HN! I'm one of the maintainers of `llm`. I've been working alongside a trusty group of contributors to bring this project to life, and we're now at a point where we're ready to share it with the world. Large language models (LLMs) are taking the computing world by storm due to their emergent abilities that…
Does a similar job
all alternatives →
My LLM CLI tool can run tools now, from Python code or plugins2025 · simonwillison.net · ▲529LLM 0.26 is out with the biggest new feature since I started the project: support for tools. You can now use the LLM CLI tool—and Python library—to grant LLMs from …
LLM-Term – Simple Rust-based CLI assist tool2024 · github.com · ▲32Current AI-assisted CLI tools are often part of larger systems and work better on Linux. I built llm-term to address these. It's a Rust-based tool that compiles into a single binary file. You only need to download the binary, add it to your PATH, and configure your OpenAI key to get started. While llm-term offers an option for gpt-4o, it works great with gpt-4o-mini. So it's not costly. I appreciate any feedback or suggestions.
Tiny-vLLM – high performance LLM inference engine in C++ and CUDAMay 2026 · github.com · ▲205Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM - jmaczan/tiny-vllm
TensorZero2025 · github.com · ▲307Hi HN! We're Gabriel & Viraj, and we're excited to open source TensorZero. To be a little cheeky, TensorZero is an open-source platform that helps LLM applications graduate from API wrappers into defensible AI products. 1. Integrate our model gateway 2. Send metrics or feedback 3. Unlock compounding improvements in quality, cost, and latency It enables a data & learning flywheel for LLMs by unifying: • Inference: one API for all LLMs, with <1ms P99 overhead • Observability: inference & feedback → your database • Optimization: better prompts, models, inference strategies • Experimentation:…
Ollama – Run LLMs on your Mac2023 · github.com · ▲284Hi HN A few folks and I have been working on this project for a couple weeks now. After previously working on the Docker project for a number of years (both on the container runtime and image registry side), the recent rise in open source language models made us think something similar needed to exist for large language models too. While not exactly the same as running linux containers, running LLMs shares quite a few of the same challenges. There are "base layers" (e.g. models like Llama 2), specific configuration to run correctly (parameters, temperature, context window sizes etc). There's…
Questions this answers
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 23d ago · stillwet.art
FastRouter.ai▲364Route requests to the right LLM for cost, latency & quality
AI · 24d ago · fastrouter.ai
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, May 2023
the whole month →
In today's world, catchy headlines and articles often distract readers from the facts and relevant information. By utilizing OpenAI's language models, Boring Report processes sensationalist news articles, transforms them into the content you see, and helps readers focus on the essential details. We recently updated our iOS app experience, so any and all feedback would be appreciated. App Link: https://apps.apple.com/us/app/boring-report-news-by-ai/id644...
AI · 2023 · boringreport.org
Momentum Page▲834Launch your website in seconds, get users in minutes
Dev tools · 2023 · page.mmntm.build
