Sipp – Run small local LLMs in browser 3x faster
Hi HN! Sipp is an open-source AI inference library for running local models in browsers with up to 3x faster decode speeds than alternative libraries. My background is in HCI (human-computer interaction) and graphics programming. Me along with my co-founder have been experimenting and thinking a lot about what the next user experience will look like when tokens are commodified to the point of being essentially “free.” A motivation for us was to try to move beyond the chat app and information retrieval use cases that are dominant now, and figure out how AI could instead act as a continuous…
What it does
In the maker’s words, at launch
Hi HN! Sipp is an open-source AI inference library for running local models in browsers with up to 3x faster decode speeds than alternative libraries. My background is in HCI (human-computer interaction) and graphics programming. Me along with my co-founder have been experimenting and thinking a lot about what the next user experience will look like when tokens are commodified to the point of being essentially “free.” A motivation for us was to try to move beyond the chat app and information retrieval use cases that are dominant now, and figure out how AI could instead act as a continuous and silent hand that helps the user indirectly, subtly monitoring their intent and dynamically generating or shifting the UI to meet their needs. In our explorations, we ran into two pain points: 1) when running AI in the browser, performance wasn’t good enough for real-time applications, and model loading and caching were issues; 2) when trying to run locally on desktop, there weren’t any good solutions for embedding a model for use through platforms like Electron or Tauri. So a few months ago, we started contributing to the llama.cpp project to explore browser inference. At the time, it didn’t have great WebGPU support. We helped get the backend working across most modern GPUs (16 bit fp is required) with fairly good ops coverage. Simultaneously, we started building up Sipp from the ground up using Rust and C++ to create a unified library around how local and cloud inference can be used together in real applications and use cases. The result, Sipp achieves up to a 3x speedup in overall token decoding compared to alternative libs. But we believe there is more room for improvement here. Our goal is to investigate more bespoke inference pipelines per model architecture, which will allow us to further optimize for both compute and memory constraints more effectively. Sipp uses a single, unified client API. A large concern for us was how to create a library that effectively bridges both local and cloud inference in a simple way. We wanted developers to start by running a small local model in the browser via WebGPU (or via other backends), and then scale the exact same code path to a self-hosted gateway (CUDA/Vulkan/Metal) or a cloud provider just by changing the endpoint. This enables the benefits of running locally for small tasks while letting you offload to a provider or cloud for tasks that require more intelligence. Currently, our primary focus has been optimizing the browser experience, but we are actively working on creating additional backends for running an LLM locally via our client API, which works across Node, Rust, and Python right now. What we are most excited about next is pushing our existing backends even further. As anyone working with a high-performance system knows, raw algorithms and efficient matmul are only half the battle. A significant portion of bottlenecks in real-time systems comes from inefficient memory management and VRAM<>RAM transfer costs. We believe there is a massive opportunity to extract even more performance through aggressive kernel fusion. By creating “bespoke” kernels tailored to specific model architectures, we can drastically minimize intermediate memory copies. Our next goal is to see exactly how close we can push local inference to the theoretically “perfect” decode and prefill speeds for consumer hardware. We have a live, simple chat demo running entirely on-device on our site, as well as a benchmarking tool if you want to test or verify the performance differences on your own hardware. We’d love for you to tear into the code, run the benchmarks, and tell us where we can improve. I’ll be here all day to answer any questions about the architecture, kernel fusion, or HCI in the age of local LLMs. More in depth tech info is here: https://dev.to/constant_chen_/sipp-a-local-first-runtime-for...
Does the same job
all alternatives →
Whisper Internet Infra AI ContextMay 2026 · ▲95Free MCP for security AI: live BGP, DNS, threat graph
- BOBhumi–OSS Python Library w Rust Underhead for 2.5x Faster LLM Inference2025 · bhumi.trilok.ai · ▲8
Read the full blogpost at https://rach.codes/blog/Introducing-Bhumi (click on reader to see the technical breakdown!) AI inference should be fast, but in practice it’s painfully slow. Inference bottlenecks slow down LLM-powered chatbots and AI workflows everywhere. I built Bhumi to fix that. Bhumi is a Python library designed for developers, yet its performance-critical core is implemented in Rust (via PyO3) for near-native speed. This hybrid approach delivers up to 2.5x faster response times across providers like OpenAI, Anthropic, and Gemini—without changing the…
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 27d ago · cactuscompute.com
Launched alongside, June 2026
the whole month →
Fundraisly▲1,544AI fundraising agent that finds investors and books meetings
AI · Jun 2026 · fundraisly.com
- H6Homebrew 6.0.0▲1,481
Today, I’m proud to announce Homebrew 6.0.0. The most significant changes since 5.1.0 are a new tap trust security mechanism, the new faster, smaller, default internal Homebrew JSON API, sandboxing on Linux, better defaults informed by our user survey, many brew bundle improvements, improved performance and initial support for macOS 27 (Golden Gate). Happy to discuss any questions here!
Dev tools · Jun 2026 · brew.sh
- PU
hope you enjoy
Life & fun · Jun 2026 · vorpus.github.io
- IM
Life & fun · Jun 2026 · hackernewstrends.com