Ferrum

What it does

Ferrum is an open-source, Rust-native runtime for running and serving local language models on Apple Silicon Metal and NVIDIA CUDA. One binary gives you an interactive CLI plus OpenAI-compatible Chat Completions and Responses APIs, without requiring Python, PyTorch, or vLLM at runtime. Choose a model explicitly, run it locally, or expose the same model over HTTP.

Run and serve local LLMs with one Rust binary. Ferrum provides Apple Silicon Metal and NVIDIA CUDA acceleration plus OpenAI-compatible APIs without a Python runtime.

Ferrum runs language models on Apple Silicon Metal and NVIDIA CUDA, then serves them through an OpenAI-compatible API — without requiring Python, PyTorch, or vLLM at runtime. Ferrum keeps the first experience small: inspect the install, name a model explicitly, run a prompt, or expose the same model over HTTP. A single CLI and server binary with no Python runtime in the supported prebuilt release path. Apple Silicon uses Metal and GGUF; NVIDIA sm89 uses CUDA with GPTQ or safetensors models. Chat Completions, streaming usage, Responses text, function tools, and structured output. Ferrum never silently selects a model. Every run and server starts from a model you named. Continuous batching,…from ferrum.pandaailabs.com

• 
  ChattyUI2024 · ▲149

  Run open-source LLMs locally in the browser using WebGPU

• FJ
  Ferrum.js – Rust features in JS: traits and advanced iteration2019 · github.com · ▲63
• 
  Unsloth Desktop25d ago · unsloth.ai · ▲222

  Run and train AI models locally on your desktop

• 
  IronClawFeb 2026 · ▲234

  Secure, open-source alternative to OpenClaw

• 
  npm i -g hotcellAug 2026 · github.com · ▲97

  Local sandboxes for AI agents on your Mac, Linux, bare metal

• 
  Second State VM2020 · ▲78

  High performance secure & portable Rust functions in Node.js

• 
  I trained a 125M model to autocomplete piano on-device▲598

  I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.

  AI · 17d ago · simedw.com

• 
  Astute▲585

  Automate your B2B brand going viral, with new media creators

  AI · 18d ago · company-app.joinastute.com

• 
  Grok Bot▲547

  AI teammates that you can give real work to

  AI · 25d ago · x.ai

• 
  Needle2: 14MB agentic LLM for phones, wearables, smart home and robots▲537

  Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…

  AI · 26d ago · cactuscompute.com

• 
  Coldtea.ai▲512

  Make your software self-driving

  AI · 30d ago · coldtea.ai

• 
  Soloop▲472

  Approval-first Agent OS for solo founders

  AI · 30d ago · soloop.io

• 
  Open-Source eInk Bike Computer▲389

  OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.

  Dev tools · 1d ago · opentrailpaper.com

• 
  Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s▲237

  Run Qwen3.8-Flash-Next (125B MoE, 104 GB at 4-bit) on Macs with a fraction of that RAM by streaming experts from SSD. MLX + Swift, Ollama-compatible API. - carloslfu/slotstream

  Dev tools · 4d ago · github.com

• 
  TERMy – A fast terminal assistant that does not use LLMs▲188

  NPC-Forge is a framework for building conversational agents that run on the CPU without relying on machine learning or LLMs. - NPC-Forge/docs/development.md at main · gioblu/NPC-Forge

  AI · 2d ago · github.com

• 
  Weedout – Safari extension that hides YouTube AI-labeled videos▲182

  Weedout is a Safari extension that filters out videos YouTube labels as “Made with AI”.

  AI · 4d ago · masteranza.github.io

• 
  HN Match Maker – Matching "Who Wants to Be Hired?" With "Who's Hiring?"▲123

  Find your next hire or your next role from Hacker News monthly threads. AI-powered matching between candidates and job postings.

  AI · 4d ago · hnmatchmaker.com

• 
  OwnTime – a chess clock for your day's priorities▲111

  I made OwnTime to balance between competing priorities in my life. The idea is based on two main influences: the concept of "roles" from "The 5 Choices" (2015), and my repeated failure to effectively implement time blocking due to the necessary flexibility in my role. The app allows you to define time budgets, which are not much more than a few mutually exclusive countdown timers. The UX is essentially that of a chess clock for an arbitrary number of players with configurable time. The whole point of the app is to kick you out of the running role/priority when its time is up. One…

  Life & fun · 4d ago · owntime.app