A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo)
A 3.16M-parameter INT4 transformer running entirely in the on-chip memory of a Xilinx Kria KV260. Zero DRAM in the token loop, 59,965 tok/s on the fabric, bit-exact. Chat with it live.
A tiny LLM running at 21,000 tok/s on a $250 FPGA (Live Demo) launched on August 10, 2026 with 79 votes, #517 of 23,274 launches that month and more than 95% of that year's launches.
In plain words
A 3.16-million-parameter language model runs entirely on the on-chip memory of a $250 Xilinx Kria KV260 FPGA, achieving 59,965 tokens per second without using external DRAM. The INT4 transformer operates as a live chatbot, with the model's weights and inference computations contained fully within the FPGA's reconfigurable logic. This project demonstrates that practical language model inference is possible on consumer-grade hardware through efficient quantization and specialized hardware deployment.
written from the facts on this page · September 2026
From the sources
A 3.16M-parameter INT4 transformer running entirely in the on-chip memory of a Xilinx Kria KV260. Zero DRAM in the token loop, 59,965 tok/s on the fabric, bit-exact. Chat with it live. I was so impressed by the chatjimmy.ai demo by Taalas, I wanted to see what I could squeeze inside the fabric fabric: The FPGA's reconfigurable logic. Same thing as 'PL'. Where the whole model runs here. of a $250 FPGA FPGA: Field-Programmable Gate Array: a chip full of reconfigurable logic you wire into a custom digital circuit, instead of running software on a fixed CPU. . By not using the (4 GB available) DDR DDR: The off-chip DRAM (the board's main memory), ~20 GB/s, shared by both the CPU and the fabric.…from mikeayles.com
Does a similar job
all alternatives →
I built a tiny LLM to demystify how language models workApr 2026 · github.com · ▲915Built a ~9M param LLM from scratch to understand how they actually work. Vanilla transformer, 60K synthetic conversations, ~130 lines of PyTorch. Trains in 5 min on a free Colab T4. The fish thinks the meaning of life is food. Fork it and swap the personality for your own character.
Kitten TTS – 25MB CPU-Only, Open-Source TTS Model2025 · github.com · ▲1,003Kitten TTS is an open-source series of tiny and expressive text-to-speech models for on-device applications. We are excited to launch a preview of our smallest model, which is less than 25 MB. This model has 15M parameters. This release supports English text-to-speech applications in eight voices: four male and four female. The model is quantized to int8 + fp16, and it uses onnx for runtime. The model is designed to run literally anywhere eg. raspberry pi, low-end smartphones, wearables, browsers etc. No GPU required! We're releasing this to give early users a sense of the latency and voices…
Tiny-vLLM – high performance LLM inference engine in C++ and CUDAMay 2026 · github.com · ▲205Build your own high performance LLM inference engine in C++ and CUDA - a smaller version of vLLM - jmaczan/tiny-vllm
Z80-μLM, a 'Conversational AI' That Fits in 40KBDec 2025 · github.com · ▲514How small can a language model be while still doing something useful? I wanted to find out, and had some spare time over the holidays. Z80-μLM is a character-level language model with 2-bit quantized weights ({-2,-1,0,+1}) that runs on a Z80 with 64KB RAM. The entire thing: inference, weights, chat UI, it all fits in a 40KB .COM file that you can run in a CP/M emulator and hopefully even real hardware! It won't write your emails, but it can be trained to play a stripped down version of 20 Questions, and is sometimes able to maintain the illusion of having simple but terse conversations…
Fine-tune an 8B model on a 4 GB laptop GPUAug 2026 · github.com · ▲139Fine-tune LLMs from one YAML. Layer streaming trains an 8B model on a 4 GB laptop GPU. - MakazhanAlpamys/Soup
Explore large language models with 512MB of RAM2023 · github.com · ▲138Explore large language models in 512MB of RAM. Contribute to jncraton/languagemodels development by creating an account on GitHub.
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 22d ago · stillwet.art
FastRouter.ai▲364Route requests to the right LLM for cost, latency & quality
AI · 23d ago · fastrouter.ai
Source: Show HN post ↗
Launched alongside, August 2026
the whole month →- TL
Life & fun · Aug 2026 · louisabraham.github.io
Clipto MCP▲655Hey HN, We recently built Clipto. It’s a tool that lets you search over terabytes of video, audio, and images on your computer, without relying on the cloud. Motivation: we probably all had this similar experience, we know a moment exists in a video or audio, but finding it takes hours scrubbing the timeline. You can send all the media to process in the cloud, but it’s slow, expensive and raises privacy concerns. So we decided to build our own on-device media search engine. How it works (high level): 1. We ingest video, audio, image; normalize formats via ffmpeg; run content analysis to…
Work · Aug 2026 · clipto.com
Hello HN! I found that picking out plausible but diverse skin tones for my digital art and game development projects was kind of difficult, and I got curious about if there was a way to define a color space that made it easy. I've built a color picker and procedural generation algorithm based on the space as well as a bunch of other fun js features and demos throughout the page that use the equations. If you find it interesting, I have lots of explanations of how I built it and what properties the space has. The methodology might be a bit shaky, but hopefully the result is as helpful for…
Life & fun · Aug 2026 · toneyalexander.github.io
Astute▲606Automate your B2B brand going viral, with new media creators
AI · Aug 2026 · company-app.joinastute.com