Serve 100 Large AI models on a single GPU with low impact to TTFT
I wanted to build an inference provider for proprietary AI models, but I did not have a huge GPU farm. I started experimenting with Serverless AI inference, but found out that coldstarts were huge. I went deep into the research and put together an engine that loads large models from SSD to VRAM up to ten times faster than alternatives. It works with vLLM, and transformers, and more coming soon. With this project you can hot-swap entire large models (32B) on demand. Its great for: Serverless AI Inference Robotics On Prem deployments Local Agents And Its open source. Let me know if anyone…
Serve 100 Large AI models on a single GPU with low impact to TTFT launched on November 8, 2025 with 7 votes, #1,843 of 9,796 launches that month and more than 75% of that year's launches.
What it does
Contribute to leoheuler/flashtensors development by creating an account on GitHub.
A blazing-fast inference engine that loads models from SSD to GPU VRAM up to 10x faster than alternative loaders. Hotswap large models in < 2 seconds . Acknowledgement: This project includes substantial code originally developed by ServerlessLLM , used under the Apache License, Version 2.0. Traditional model loaders slow down your workflow with painful startup times. flashtensors is built on top of ServerlessLLM storage library to eliminate bottlenecks and maximize performance. The result: An inference engine that scales by usage not by model. Our Goal is to make it dead simple to run large models on any device without having to wait minutes for startup times. Flashtensors offers a simple…from github.com
In the maker’s words, at launch
I wanted to build an inference provider for proprietary AI models, but I did not have a huge GPU farm. I started experimenting with Serverless AI inference, but found out that coldstarts were huge. I went deep into the research and put together an engine that loads large models from SSD to VRAM up to ten times faster…
Does a similar job
all alternatives →
General ComputeMay 2026 · generalcompute.com · ▲315AI models that run on an inference cloud optimized for speed
Cerebrium2024 · cerebrium.ai · ▲300If you haven’t noticed, GPUs are scarce right now. The global boom in inference has demand growing faster than supply can keep up, and at Cerebrium ...
We made our own inference engine for Apple Silicon2025 · github.com · ▲186We wrote our inference engine on Rust, it is faster than llama cpp in all of the use cases. Your feedback is very welcomed. Written from scratch with idea that you can add support of any kernel and platform.
- IBI built a toy TPU that can do inference and training on the XOR problem2025 · tinytpu.com · ▲134
We wanted to do something very challenging to prove to ourselves that we can do anything we put our mind to. The reasoning for why we chose to build a toy TPU specifically is fairly simple: - Building a chip for ML workloads seemed cool - There was no well-documented open source repo for an ML accelerator that performed both inference and training None of us have real professional experience in hardware design, which, in a way, made the TPU even more appealing since we weren't able to estimate exactly how difficult it would be. As we worked on the initial stages of this project, we…
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 21d ago · stillwet.art
FastRouter.ai▲364Route requests to the right LLM for cost, latency & quality
AI · 23d ago · fastrouter.ai
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, November 2025
the whole month →
Guideflow▲1,338Create interactive product demos and step-by-step guides in seconds with Guideflow, the leading AI-powered demo automation tool.
AI · Nov 2025 · guideflow.com
Hardware and software projects by Ali Roberts
Life & fun · Nov 2025 · bitsnpieces.dev
Welltory▲1,028Measure your wellness. Use data to feel great every day.
Work · Nov 2025 · welltory.com
TrustMRR▲833LLM Gateway is a b2b artificial intelligence based in United States, founded in 2025. $259,539/month in verified Stripe revenue. $78,764 MRR. Ahrefs Domain Rating 52/100. growing 55% month-over-month. Track LLM Gateway's verified metrics on TrustMRR.
Growth · Nov 2025 · trustmrr.com
Boing▲782Life & fun · Nov 2025 · boing.greg.technology