Alternatives
Products that do what Shimmy v2.0 does
The first pure-Rust GGUF inference engine. No C. No Python.
- 1WM
We wrote our inference engine on Rust, it is faster than llama cpp in all of the use cases. Your feedback is very welcomed. Written from scratch with idea that you can add support of any kernel and platform.
2025 · github.com
- 2OS
Hi HN, I built a specialized inference engine for running 4-bit Gemma 4 26B-A4B-IT on any M-series Mac using about 2 GB of RAM. It is called TurboFieldfare and is written in Swift and Metal. I have always adored on-device AI. It feels like magic that you can run a powerful NN on your Mac or iPhone. So I wanted to push the limits a bit and run a model whose weights don’t fit in memory. The model’s 4-bit quantized weights occupy roughly 14 GB, which makes running it with conventional inference tools almost impossible on an 8 GB or even 16 GB Mac once the OS, applications, and KV cache are…
Jul 2026 · github.com
- 3FL
I've been playing around with https://github.com/zphang/minimal-llama/ and https://github.com/tloen/alpaca-lora/blob/main/finetune.py, and wanted to create a simple UI where you can just paste text, tweak the parameters, and finetune the model quickly using a modern GPU. To prepare the data, simply separate your text with two blank lines. There's an inference tab, so you can test how the tuned model behaves. This is my first foray into the world of LLM finetuning, Python, Torch, Transformers, LoRA, PEFT, and Gradio. Enjoy!
2023 · github.com
- 4RR
runNburn is an Apache-2.0 Rust inference engine for quantized GGUF models that are too big for your fast memory. The core idea: weights stay file-backed (mmap), host residency stays under an explicit byte budget (--ram-budget), and GPU caches are sized from detected free/total VRAM — never from device-name presets. There is no conversion step, no sidecar cache files, no silent requantization. The GGUF on disk is the single source of truth. The result that made me want to post this: Tencent's Hy3 (295B total / 21B active sparse MoE, a single 97.8 GiB Q2_K GGUF) runs on my desktop…
Jul 2026 · github.com
- 5CR
Jun 2026 · github.com
- 6CF
Oct 2025 · github.com
- 7WM
Try it out! https://glhf.chat/ Hey HN! We’ve been working for the past few months on a website to let you easily run (almost) any open-source LLM on autoscaling GPU clusters. It’s free for now while we figure out how to price it, but we expect to be cheaper than most GPU offerings since we can run the models multi-tenant. Unlike Together AI, Fireworks, etc, we’ll run any model that the open-source vLLM project supports: we don’t have a hardcoded list. If you want a specific model or finetune, you don’t have to ask us for it: you can just paste the Hugging Face link in and…
2024 · glhf.chat
- 88F
Hi HN! I'm just sharing a project I've been working on during the LLM Efficiency Challenge - you can now finetune Llama with QLoRA 5x faster than Huggingface's original implementation on your own local GPU. Some highlights: 1. Manual autograd engine - hand derived backprop steps. 2. QLoRA / LoRA 80% faster, 50% less memory. 3. All kernels written in OpenAI's Triton language. 4. 0% loss in accuracy - no approximation methods - all exact. 5. No change of hardware necessary. Supports NVIDIA GPUs since 2018+. CUDA 7.5+. 6. Flash Attention support via Xformers. 7. Supports 4bit and 16bit…
2023 · github.com
- 9GR
Hi everyone, wanted to share about gline-rs, an inference engine for GLiNER models written in Rust. This family of lightweight language models proved to be efficient at zero-shot Named Entity Recognition (NER) and other tasks such as Relation Extraction, while consuming less resources than large generative models (LLMs). This implementation has been written from the ground up in Rust, and supports both span- and token-oriented variants (for inference only). The goal is to provide a production-grade and user-friendly API in a modern and safe programming language, including a clean and…
2025 · github.com
- 10IV
This is a GPU "software" raytracer (i.e. using manual ray-scene intersections and not RTX) written using the WebGPU API that renders glTF scenes. It supports many materials, textures, material & normal mapping, and heavily relies on multiple importance sampling to speed up convergence.
2024 · github.com
- 11LO
Hi HN, I’m Joe. My friends Matthew, Jake and I are building Luminal (https://luminalai.com/), a GPU compiler for automatically generating fast GPU kernels for AI models. It uses search-based compilation to achieve high performance. We take high level model code, like you'd have in PyTorch, and generate very fast GPU code. We do that without using LLMs or AI - rather, we pose it as a search problem. Our compiler builds a search space, generates millions of possible kernels, and then searches through it to minimize runtime. You can try out a demo in `demos/matmul` on mac to…
2025 · github.com
- 12
An ultra-fast, single-binary MCP server written in Rust as a lightweight alternative to Node.js/Python. - StamManif/mcp-stama
24d ago · github.com
- 13LC
Complete llama.cpp tutorial for 2026. Install, compile with CUDA/Metal, run GGUF models, tune all inference flags, use the API server, speculative decoding, and benchmark your hardware. https://vucense.com/dev-corner/llama-cpp-tutorial-run-gguf-m...
Apr 2026
- 14AR
I’ve been experimenting with Rust lately and wanted a project that would help me explore some of its lower-level and performance-oriented features. Inspired by Sebastian Lague’s videos, I decided to implement my own ray tracer from scratch. The initial goal was just to render a simple 3D scene in the browser at a reasonable frame rate. It evolved into a small renderer that can: • Run locally or on the web using wgpu and WebAssembly • Perform mesh rendering with a Bounding Volume Hierarchy (BVH) for acceleration • Simulate both direct and indirect illumination for photorealistic results • Be…
Nov 2025 · github.com
- 15AT
2017 · github.com
- 16S5
2025 · github.com
- 17
High performance storage engine for efficient LLM inference and GPU Training.
20h ago · theopenlake.com
- 18SA
Hi HN, I’ve been working on Shibuya, a next-generation Web Application Firewall (WAF) built from the ground up in Rust. I wanted to build a WAF that didn't just rely on legacy regex signatures but could understand intent and perform at line-rate using modern kernel features. What makes Shibuya different: Multi-Layer Pipeline: It integrates a high-performance proxy (built on Pingora) with rate limiting, bot detection, and threat intelligence. eBPF Kernel Filtering: For volumetric attacks, Shibuya can drop malicious packets at the kernel level using XDP before they consume userspace resources.…
Feb 2026 · ghostklan.com
- 19HG
Tabs, splits, and tmux work fine until you have several projects open with logs, tests, and long-running shells. I kept rebuilding context instead of resuming work. Horizon puts shells on an infinite canvas. You can arrange them into workspaces and reopen later with layout, scrollback, and history intact. Built in 3 days with Claude/Codex, dogfooding the workflow as I went. Feedback and contributions welcome.
Mar 2026 · github.com
- 20
A fast, rootless sandbox and virtual resource runtime for any workload, including untrusted and AI-generated code. Daemonless: a real, kernel-enforced container in ~3.5 ms from an OCI image, out of one static binary - getkern/kern
13d ago · github.com
- 21M3
Grant Sanderson (3Blue1Brown) created Manim, the Python library he uses to make the math animations in his videos. We reimplemented Manim with the same Python API, but the implementation underneath is Rust, connected to Python through PyO3. The Rust code uses wgpu, so rendering happens on the GPU. To run it in the browser, we compiled the Rust parts to WebAssembly so the PyO3 extension loads in Pyodide. In the browser, wgpu targets the WebGPU API, so animations render in real time on your GPU through the browser. The editor is Monaco (the editor that powers VS Code) with a live preview:…
Jul 2026 · studio.academa.ai
- 22RF
Alternative to magic.h and infer. Zero dependencies. Fully extensible. Works in no_std, async, and embedded contexts.
2025 · crates.io
- 23IM
My cofounder and I run a startup in oncology, where we handle cancer genomics data. It occurred to me that, thanks to a recent complexity theory result, there's a clever way to run bioinformatics algorithms using far less RAM. I built this Rust engine for running whole-genome workloads in under 100MB of RAM. Runtime is a little longer as a result - O(TlogT) instead of O(T). But it should enable whole-genome analytics on consumer-grade hardware.
Nov 2025 · github.com
- 24SO
Hello HN! I'm an engineer at Splitgraph and recently started learning Rust so I could make my first contribution to Seafowl [0], an early stage analytical database. Along the way I figured out a database hosting hack on GCP and wanted to share it with HN. It's a way to achieve "true" scale to zero database hosting that could be useful for certain side projects or spiky traffic situations. A recurring problem I've faced with side projects is the need for Postgres, but no desire to deploy or maintain new instances. So when I learned GCP's "always free" tier includes serverless [1] I got…
2023 · github.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →