Alternatives
Products that do what I run 30B 22tok/s, 109tok/s not novel,6GB/16GB RAM overcoming llama.cpp does
Democratisation of local AI is key. I've been working on pushing the limits of commercial hardware, squeezing any extra bit possible. My Scientific Agentic AI hareness helped me to reallocate every single bit of it. I rewrote the Kernel, I went down the CUDA rabbit hole until I have been able to explain any bit and any ms of computational power involved in the process pushing the Qwen 30B-A3B from 8 tok7s to 19 tok/s with llama.cpp up to 22.2 tok/s with my project and 109 tok/s on not novel content and speeding up the prefill by 5-9X
- 18F
Hi HN! I'm just sharing a project I've been working on during the LLM Efficiency Challenge - you can now finetune Llama with QLoRA 5x faster than Huggingface's original implementation on your own local GPU. Some highlights: 1. Manual autograd engine - hand derived backprop steps. 2. QLoRA / LoRA 80% faster, 50% less memory. 3. All kernels written in OpenAI's Triton language. 4. 0% loss in accuracy - no approximation methods - all exact. 5. No change of hardware necessary. Supports NVIDIA GPUs since 2018+. CUDA 7.5+. 6. Flash Attention support via Xformers. 7. Supports 4bit and 16bit…
2023 · github.com
- 2SU
Here's a project I've been working on for the last few months. It's a new (I think) algorithm, that allows to adjust smoothly - and in real time - how many calculations you'd like to do during inference of an LLM model. It seems that it's possible to do just 20-25% of weight multiplications instead of all of them, and still get good inference results. I implemented it to run on M1/M2/M3 GPU. The mmul approximation itself can be pushed to run 2x fast before the quality of output collapses. The inference speed is just a bit faster than Llama.cpp's, because the rest of implementation…
2024 · asciinema.org
- 3L3
Hi everyone, I'm kinda involved in some retrogaming and with some experiments I ran into the following question: "It would be possible to run transformer models bypassing the cpu/ram, connecting the gpu to the nvme?" This is the result of that question itself and some weekend vibecoding (it has the linked library repository in the readme as well), it seems to work, even on consumer gpus, it should work better on professional ones tho
Feb 2026 · github.com
- 4WM
We wrote our inference engine on Rust, it is faster than llama cpp in all of the use cases. Your feedback is very welcomed. Written from scratch with idea that you can add support of any kernel and platform.
2025 · github.com
- 5RA
Hi there, looking for feedback on my new project "Featherless.AI" The idea is to allow users to run all the models on hugging face instantly. Via the OpenAI API compatible endpoint. Why? Because its a real chore to download models and spin up GPUs, especially if you want to test multiple models. Not to mention GPUs cost multiple dollars an hour to rent. And if we want more people to use open source AI, we got to make it easier for them to try and play with all of them. So what if instead of spinning up dedicated GPUs per model (which is what every provider is doing) We can startup a LLM…
2024 · featherless.ai
- 6NA
I've been tired with the current options on the market for awhile and decided to do something about it after the running into the disaster that is MLOps at my last two startups and having to manage a multiple operation platforms both for my fellow ML engineers, the general application CI/CD and orchestration layers while simultaneously building the application itself. Its still extremely early for the product but its functioning and is well on its way. I'd love feedback on the approach and peoples thoughts on the problem space. Personally my irritations have been in the poor tooling,…
2025 · egdaemon.com
- 7
I made this after seeing someone posit the idea online yesterday over lunch then spent some time refining it. So far it's pretty impressive IMO! Right now I am running Qwen3-30B-A3B on my 24gb unified memory m4 MacBook Pro at 50 tok/sec and this should definitely not be working for such a large model on my middling hardware. Things are detailed in the README to get up and running and DESIGN.md has details on all the choices and such made along the way.
23d ago · github.com
- 8IB
You know that old TI calculator you used in high school, then put in a box and forgot about? Have you ever wished you had an operating system for your calculator with preemptive multitasking, dynamic memory management, a tree filesystem conforming to the FHS, and all the comforts of Unix? Well, good news: that's totally a thing that exists. I've been working on my kernel for about three and a half years now and I'm looking for new contributors to help out. It's written entirely in z80 assembly, and it's both challenging and fun to work on. There's an IRC channel for contributors or people…
2014
- 9RS
Hi HN! I've been hacking on this side project for the last month or two with the goal of making it dead simple to use cloud GPUs. I ran into this problem personally during the phd, and built my own tooling around it. I always thought it'd be fun to try to turn that tooling into a more general product... and bitbop.io is the result! All you have to do is run `ssh bitbop.io`, and you get your own personal dev GPU workstation in the cloud. Looking forward to hearing your thoughts!
2024 · twitter.com
- 10IB
I had 14,000 photos sitting on a drive and wanted an excuse to play with local vision models and Elixir/Phoenix. I originally tried to get LLaVA to tell me if a photo was 'good' or matched my style, but quickly learned that LLMs have terrible taste. I ended up demoting the LLM to just extract metadata, and built a custom CLIP/Ridge Regression pipeline to actually learn my preferences based on how I rate things. The stack is Phoenix/Oban on the orchestrator side, and Python/FastAPI/Instructor for the AI workers. Happy to answer any questions about the architecture,…
Apr 2026 · qwelian.com
- 11WB
Over the past few months, as we scaled our internal AI Agents, we hit a dead end: Running LLM-generated arbitrary code in Docker is basically running naked on security due to container escape risks. But using full traditional VMs takes minutes to boot and eats too much memory to support high-density concurrency. We loved the developer experience of SaaS sandboxes on the market, but they are closed-source, expensive, and have too high a barrier to entry for self-hosting. So, our team decided to build our own. After months of grinding, using RustVMM and KVM, we built a blazing-fast,…
Apr 2026 · github.com
- 12AL
Hello all, a couple of moons ago I ported karpathy's llama2.c code to run inference on the TinyStories 260K & 15M checkpoints on the on the PS Vita with the ability to download/delete the models on device. Runs showed that the 260K model ran at ~120 tok/s and at 15M ran at 1.8 tok/s, which probably could be a bit higher if it weren't a single threaded application. Had fun working on it as a weekend project, check it out for yourselves: https://github.com/callbacked/psvita-llm
2025 · github.com
- 135L
We've built InferX, a specialized runtime environment that fundamentally changes how LLMs are served. The core problem we solve is the latency bottleneck in AI inference, especially with large models. Current systems waste resources or suffer from painfully slow cold starts. InferX's AI-native architecture, with its "snapshot" technology, enables: * *Sub-2s cold starts:* Spin up models instantly. * *High density:* Serve more LLMs on the same GPUs. * *Optimal efficiency:* Maximize GPU utilization. This isn't just another API; it's a new execution layer designed from the ground up for the…
2025 · github.com
- 14IB
Built a simple web app that tells you which open-source LLMs will work on your hardware. It auto-detects your specs, shows compatible models from Hugging Face, gives realistic performance estimates (tokens/sec), and recommends quantization settings. You can also manually input specs to see "what if I upgraded my RAM?" Made this after wasting time downloading giant models only to find they crawled on my hardware. Hope it saves you some frustration!
2025 · caniusellm.com
- 15FG
Built this as a guide for users of my hosting service. Everything here was run on a real instance. The Mesa/LLVM dependency is genuinely painful on a 1GB disk, and I couldn't find a way around it. WindowMaker won the window manager battle purely on dependency chain cleanliness; it pulls in no GTK, no Python, no audio stack. Happy to answer questions about any of the choices made. Screenshots at the end.
Mar 2026 · tierhive.com
- 16IM
This is a personal project I've been working on and off for the past few years. It's a set of tools that have allowed me to prototype quickly small (and increasingly bigger) kernels, and create userspace programs to interact with them. Supports riscv64, amd64 and i386. The intel port has been tested and used on real hardware, from big dual-socket Xeon machines to an old X220. It is all C, I have plan to make rust bindings for the kernel library. But again, they're plans at this stage. Porting to new architectures is relatively simple, a basic port to riscv took me a couple of weeks of…
2024 · github.com
- 17S1
I wanted to build an inference provider for proprietary AI models, but I did not have a huge GPU farm. I started experimenting with Serverless AI inference, but found out that coldstarts were huge. I went deep into the research and put together an engine that loads large models from SSD to VRAM up to ten times faster than alternatives. It works with vLLM, and transformers, and more coming soon. With this project you can hot-swap entire large models (32B) on demand. Its great for: Serverless AI Inference Robotics On Prem deployments Local Agents And Its open source. Let me know if anyone…
Nov 2025 · github.com
- 18IB
I've been experimenting with ways to increase AI adoption for non-technical people. Basically, all companies are pushing for AI because it's all over the news and they feel left behind but most people have no clue where to start. I think 90% of people (ie non coders) are sufficiently well served by using cowork instead of claude code or something similar. If we can get people from sales, customer support, marketing, etc to collaborate with skills and cowork to form a company brain, I think it's gold. So I think there's opportunity for the community to share skills that work well for 1000s of…
Jun 2026 · claudinho.xyz
- 19SY
Hey HN, If you tried running open-source models like Llama 3.1 70B or 405B, you might have noticed that it gets very expensive. The reason looks obvious enough that you might have stopped even before trying it! - GPUs are very expensive to buy or rent - Running the most performing LLMs need 4, 8 or even 16 top of the line Nvidia GPUs - And that won’t get you anywhere near the level of VRAM needed to batch enough to get a decent throughput and efficiency Some have even questioned if open-source LLM providers are not doing some shenanigans to provide the prices they offer. VC funded…
2024
- 20IM
Hey HN! Thank you for all the support and feedback on my original submission 2 months ago. I've been improving the backend using a MCTS/AlphaZero approach and it's currently producing much better results. My long term goal is to allow users to manage multiple projects, deployed autonomously, both from scratch and by making continual updates all prompted with natural language. The cost of each project has been lowered to $9 as performance with smaller models has improved (I migrated from Claude-3-Opus to gemini-1.5-flash). Thanks for checking it out!
2024 · saas-quick.com
- 21KP
I thought it'd be interesting to use Linux PSI (Pressure Stall Information) for an LLM runtime to trim the KV cache. This is mainly useful imo for edge devices like the Jetson Orin super nano kit which have unified memory. I haven't benched much, but plan to do so more over time and see if I can make a real use of it as I run local LLMs. Let me know if it makes sense :P (I of course vibed this idea)
Jun 2026 · github.com
- 22LS
Earlier this year, I took a month to reexamine my coding habits and rethink some past design choices. I hope to rewrite and improve my FOSS libraries this year, and I needed answers to a few questions first. Perhaps some of these questions will resonate with others in the community, too. - Are coroutines viable for high-performance work? - Should I use SIMD intrinsics for clarity or drop to assembly for easier library distribution? - Has hardware caught up with vectorized scatter/gather in AVX-512 & SVE? - How do secure enclaves & pointer tagging differ on Intel, Arm, & AMD? - What's…
2025 · github.com
- 23WB
Hey HN: Kaveh here, founder of https://www.usage.ai/ We help companies drive down AWS, GCP, and Azure spend. Why? Because the way it's done now is a pain. DevOps and Software Engineers end up spending time managing costs rather than focusing on business problems. I have been building Usage AI for almost 4 years now (4 year anniversary in 1 month from now!) with an incredible group of founding people. We started as a product just to help lower AWS EC2 costs, and now we do all major AWS services (such as RDS, OpenSearch, ElastiCache, and Redshift with more on the way) and other…
2024
- 24IM
As a handsome local AI enjoyer™ you’ve probably noticed one of the big flaws with LLMs: It lies. Confidently. ALL THE TIME. I’m autistic and extremely allergic to vibes-based tooling, so … I built a thing. Maybe it’s useful to you too. The thing: llama-conductor llama-conductor is a router that sits between your frontend (eg: OWUI) & backend (llama.cpp + llama-swap). Local-first but it should talk to anything OpenAI-compatible if you point it there (note: experimental so YMMV). LC is a glass-box that makes the stack behave like a deterministic system, instead of a drunk telling a story about…
Jan 2026
Ranked by how close each launch is in meaning, then by votes. Refine with a description →