Willow Inference Server: Optimized ASR/TTS/LLM for Willow/WebRTC/REST
Hey HN! Willow Inference Server (WIS) is a focused and highly optimized language inference server implementation. Our goal is to "automagically" enable performant, cost-effective self-hosting of released state of the art/best of breed models to enable speech and language tasks: Primarily targeting CUDA (works on CPU too) with support for low-end (cheap) devices such as the Tesla P4, GTX 1060, and up. Don't worry - it screams on an RTX 4090 too! (See benchmarks on Github). Memory optimized - all three default Whisper (base, medium, large-v2) models loaded simultaneously with TTS support…
Willow Inference Server: Optimized ASR/TTS/LLM for Willow/WebRTC/REST launched on May 23, 2023 with 13 votes, #1,634 of 4,910 launches that month and more than 75% of that year's launches. The site was still online in September 2026, 3 years on.
In plain words
Willow Inference Server is an optimized inference platform for running speech recognition, text-to-speech, and language models on self-hosted hardware. It supports GPU acceleration on CUDA devices ranging from budget options like the Tesla P4 to high-end cards, and also runs on CPU. The server can load multiple Whisper speech recognition models simultaneously within 6GB of VRAM, with additional language model and TTS capabilities available. It's designed for developers and organizations seeking cost-effective, performant alternatives to cloud-based AI services.
written from the facts on this page · September 2026
From the sources
Open source, local, and self-hosted highly optimized language inference server supporting ASR/STT, TTS, and LLM across WebRTC, REST, and WS - toverainc/willow-inference-server
In the maker’s words, at launch
Hey HN! Willow Inference Server (WIS) is a focused and highly optimized language inference server implementation. Our goal is to "automagically" enable performant, cost-effective self-hosting of released state of the art/best of breed models to enable speech and language tasks: Primarily targeting CUDA (works on…
Does a similar job
all alternatives →
Willow – Open-source privacy-focused voice assistant hardware2023 · github.com · ▲581As the Home Assistant project says, it's the year of voice! I love Home Assistant and I've always thought the ESP BOX[0] hardware is cool. I finally got around to starting a project to use the ESP BOX hardware with Home Assistant and other platforms. Why? - It's actually "Alexa/Echo competitive". Wake word detection, voice activity detection, echo cancellation, automatic gain control, and high quality audio for $50 means with Willow and the support of Home Assistant there are no compromises on looks, quality, accuracy, speed, and cost. - It's cheap. With a touch LCD display, dual…
Kitten TTS – 25MB CPU-Only, Open-Source TTS Model2025 · github.com · ▲1,003Kitten TTS is an open-source series of tiny and expressive text-to-speech models for on-device applications. We are excited to launch a preview of our smallest model, which is less than 25 MB. This model has 15M parameters. This release supports English text-to-speech applications in eight voices: four male and four female. The model is quantized to int8 + fp16, and it uses onnx for runtime. The model is designed to run literally anywhere eg. raspberry pi, low-end smartphones, wearables, browsers etc. No GPU required! We're releasing this to give early users a sense of the latency and voices…
Willow Frontier ProJul 2026 · willowvoice.com · ▲292Hey HN! We were part of YC S24, ended up pivoting around. Made dictation that worked really well, and decided to go all in on agentic. We've done two major things. 1. Scribe is a revamp of our product. Continually learns your style (can get into details if comments want). TLDR; we fingerprint based off your edits, and make dictation sound more and more like you. Post-trained Llama 3.1 8b. 2. Dictation is completely free. Unlimited unpersonalized dictation for everyone.
Moonshine Open-Weights STT models – higher accuracy than WhisperLargev3Feb 2026 · github.com · ▲316I wanted to share our new speech to text model, and the library to use them effectively. We're a small startup (six people, sub-$100k monthly GPU budget) so I'm proud of the work the team has done to create streaming STT models with lower word-error rates than OpenAI's largest Whisper model. Admittedly Large v3 is a couple of years old, but we're near the top the HF OpenASR leaderboard, even up against Nvidia's Parakeet family. Anyway, I'd love to get feedback on the models and software, and hear about what people might build with it.
Tiny-vLLM – high performance LLM inference engine in C++ and CUDAMay 2026 · github.com · ▲205Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM - jmaczan/tiny-vllm
Gan.AI TTS Model & API Playground2024 · ▲333First TTS model to support all 22 Indic languages + English
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 21d ago · stillwet.art
FastRouter.ai▲364Route requests to the right LLM for cost, latency & quality
AI · 23d ago · fastrouter.ai
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, May 2023
the whole month →
In today's world, catchy headlines and articles often distract readers from the facts and relevant information. By utilizing OpenAI's language models, Boring Report processes sensationalist news articles, transforms them into the content you see, and helps readers focus on the essential details. We recently updated our iOS app experience, so any and all feedback would be appreciated. App Link: https://apps.apple.com/us/app/boring-report-news-by-ai/id644...
AI · 2023 · boringreport.org
Momentum Page▲834Launch your website in seconds, get users in minutes
Dev tools · 2023 · page.mmntm.build
