Alternatives
Products that do what Sipp – Run small local LLMs in browser 3x faster does
Hi HN! Sipp is an open-source AI inference library for running local models in browsers with up to 3x faster decode speeds than alternative libraries. My background is in HCI (human-computer interaction) and graphics programming. Me along with my co-founder have been experimenting and thinking a lot about what the next user experience will look like when tokens are commodified to the point of being essentially “free.” A motivation for us was to try to move beyond the chat app and information retrieval use cases that are dominant now, and figure out how AI could instead act as a continuous…
- 1SU
Here's a project I've been working on for the last few months. It's a new (I think) algorithm, that allows to adjust smoothly - and in real time - how many calculations you'd like to do during inference of an LLM model. It seems that it's possible to do just 20-25% of weight multiplications instead of all of them, and still get good inference results. I implemented it to run on M1/M2/M3 GPU. The mmul approximation itself can be pushed to run 2x fast before the quality of output collapses. The inference speed is just a bit faster than Llama.cpp's, because the rest of implementation…
2024 · asciinema.org
- 2IM
Hi Hackers, Excited to share a macOS app I've been working on: https://recurse.chat/ for chatting with local AI. While it's amazing that you can run AI models locally quite easily these days (through llama.cpp / llamafile / ollama / llm CLI etc.), I missed feature complete chat interfaces. Tools like LMStudio are super powerful, but there's a learning curve to it. I'd like to hit a middleground of simplicity and customizability for advanced users. Here's what separates RecurseChat out from similar apps: - UX designed for you to use local AI as a daily driver.…
2024 · recurse.chat
- 3
- 4
- 5WM
We wrote our inference engine on Rust, it is faster than llama cpp in all of the use cases. Your feedback is very welcomed. Written from scratch with idea that you can add support of any kernel and platform.
2025 · github.com
- 6PO
Hi HN, OpenAI recently released a model for automatic speech recognition called Whisper [0]. I decided to reimplement the inference of the model from scratch using C/C++. To achieve this I implemented a minimalistic tensor library in C and ported the high-level architecture of the model in C++. The entire code is less than 8000 lines of code and is contained in just 2 source files without any third-party dependencies. The Github project is here: https://github.com/ggerganov/whisper.cpp With this implementation I can very easily build and run the model - “make…
2022 · github.com
- 7SA
2023 · github.com
- 8TV
May 2026 · github.com
- 9JA
Hello HN, I don't post on here much, but wanted to get some eyes on a new project I'm just launching. I think we definitely need one more AI code agent.. I'm a long-term C++ dev, and over 30+ years I've created some successful audio dev tools (JUCE, the Tracktion DAW, the Cmajor DSP language). All of these came from me getting annoyed with something I had to use, and deciding to have a go at my own take on whatever it was. So Juggler is my attempt at an AI code agent, after spending too many hours loving what the models could do, but hating the CLI experience, and having some opinions of…
Jul 2026 · github.com
- 10SM
We built a model router that plugs into coding agents (e.g. Claude Code, Codex, Cursor, etc.) and intelligently sends requests to the best model to serve them. Here's a quick demo of running it locally: https://www.youtube.com/watch?v=isKhAyivtfM. At Weave, we write most of our code with AI, and it's been getting more expensive. This came to a head when Opus 4.7 was released and, thanks to its tokenizer changes, our costs shot up. We knew we didn't need Opus for everything but we didn't want to lose out on the intelligence for the cases where you really need it. So we decided…
Jun 2026 · github.com
- 11IO
Hi HN! Last year the project I launched here got a lot of good feedback on creating speech to speech AI on the ESP32. Recently I revamped the whole stack, iterated on that feedback and made our project fully open-source—all of the client, hardware, firmware code. This Github repo turns an ESP32-S3 into a realtime AI speech companion using the OpenAI Realtime API, Arduino WebSockets, Deno Edge Functions, and a full-stack web interface. You can talk to your own custom AI character, and it responds instantly. I couldn't find a resource that helped set up a reliable, secure websocket (WSS) AI…
2025 · github.com
- 12
- 13
- 14VA
Voxos is an open-source desktop voice assistant that aims to put Clippy to shame while supporting new desktop workflows powered by LLMs. Tired of copy and pasting ChatGPT responses between your web browser and IDE? Does your copilot not quite do what you need it to do? I invite you to give Voxos a try and maybe even become a contributor!
2024 · gitlab.com
- 15AM
This is an open‑source Model Context Protocol (MCP) server that gives any LLM a sense of the passage of time. Most MCP demos wire LLMs to external data stores. That’s useful, but MCP is also a chance to give models perception — extra senses beyond the prompt text. Six functions (`current_datetime`, `time_difference`, `timestamp_context`, etc.) give Claude/GPT real temporal awareness: It can spot pauses, reason about rhythms, and even label a chat’s “three‑act structure”. Runs locally in <60 s (Python) or via a hosted demo. If time works, what else could we surface? - Location /…
2025 · github.com
- 16BR
Check out this impressive project that enables running LLMs entirely in the browser using WebGPU. Key features: - Zero token costs, no cloud infrastructure required - Complete data privacy through local processing - Simple 3-line code integration - Built on MLC and Transformer.js The benchmarks show smaller models can effectively handle many common tasks. Currently the project roadmap includes: - No-code AI pipeline builder - Browser-based RAG for document chat - Analytics/logging - Model fine-tuning interface
2025 · github.com
- 17AM
Hi all, I’m hacking on new features for the ClickHouse native client and wanted the same “just call the model” ergonomics JavaScript and Python now enjoy. It didn’t exist for modern C++, so I wrote one. ai‑sdk‑cpp (Apache‑2.0) gives you: - Unified calls to OpenAI (GPT‑4o) and Anthropic (Claude 3.5) with a single C++20 API. - Streaming, multi‑turn chat, error handling—all std::optional/std::variant, no macros. - Tool calling (function‑calling) so the model can hit real APIs; sync or async, runs in parallel. The tricky bit: C++ still lacks real reflection, so mapping plain functions →…
2025
- 18TF
I’d originally launched my app: Private LLM[1][2] on HN around 10 months ago, with a single RedPajama Chat 3B model. The app has come a long way since then. About a month ago, I added support for 4-bit OmniQuant quantized Mixtral 8x7B Instruct model, and it seems to outperform Q4 models at inference speed and Q8 models at text generation quality, while consuming only about 24GB of RAM[3] at 8k context length. The trick is: a) to use a better quantization algorithm and b) to use unquantized embeddings and the MoE gates (the overhead is quite small). Other notable features include many more…
2024
- 19AT
I recently built a small open-source tool to benchmark different LLM API endpoints — including OpenAI, Claude, and self-hosted models (like llama.cpp). It runs a configurable number of test requests and reports two key metrics: • First-token latency (ms): How long it takes for the first token to appear • Output speed (tokens/sec): Overall output fluency Demo: https://llmapitest.com/ Code: https://github.com/qjr87/llm-api-test The goal is to provide a simple, visual, and reproducible way to evaluate performance across different LLM providers, including…
2025 · llmapitest.com
- 20OS
Hi all! This morning, we released a new Apache 2.0 licensed model on HuggingFace for detecting hallucinations in retrieval augmented generation (RAG) systems. What we've found is that even when given a "simple" instruction like "summarize the following news article," every LLM that's available hallucinates to some extent, making up details that never existed in the source article -- and some of them quite a bit. As a RAG provider and proponents of ethical AI, we want to see LLMs get better at this. We've published an open source model, a blog more thoroughly describing our methodology (and…
2023 · vectara.com
- 21OS
Large Language Models (LLMs) are powerful, but they’re limited by fixed context windows and outdated knowledge. What if your AI could access live search, structured data extraction, OCR, and more—all through a standardized interface? We built the JigsawStack MCP Server, an open-source implementation of the Model Context Protocol (MCP) that lets any AI model call external tools effortlessly. Here’s what it unlocks: - Web Search & Scraping: Fetch live information and extract structured data from web pages. - OCR & Structured Data Extraction: Process images, receipts, invoices, and handwritten…
2025
- 22SA
2019 · schm1dt.ch
- 23R5
Hi HN, I built OpenGraviton, an open-source AI inference engine that pushes the limits of running extremely large LLMs on consumer hardware. By combining 1.58-bit ternary quantization, dynamic sparsity with Top-K pruning and MoE routing, and mmap-based layer streaming, OpenGraviton can run models far larger than your system RAM—even on a Mac Mini. Early benchmarks: TinyLlama-1.1B drops from ~2GB (FP16) to ~0.24GB with ternary quantization. At 140B scale, models that normally require ~280GB fit within ~35GB packed. Optimized for Apple Silicon with Metal + C++ tensor unpacking, plus…
Mar 2026 · github.com
- 24LT
Hi HN! I'm the author of mere.run a local first inference runtime built around an installable CLI. I believe that whenever possible we should use the stuff we already own (like our Mac laptops, decent machines gathering dust, our gaming PC) and the limited electrical power we have easy access to, like the socket in the wall next to most of us. We shouldn't have to send our data to the cloud hoping some T&C will prevent it from being used in a way that we'd regret. Most of the local AI solutions are technical, involved, and land a curious body in some package hell. People are optimizing for…
Jul 2026 · github.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →