I built a tool showing how AI providers (should) throttle their models
OP here: this project was born out of the frustration/paranoia that AI providers are throttling their models when their server load is too high. So, I set out to model and study the problem mathematically to understand what was happening, what I found was quite surprising. The idea seems natural: as the data center demand increases momentarily through the day, throttling their models (either using quantized versions, reducing the context window or lowering the tier of the model to a smaller one) seems appealing as the replacement model in principle uses less electricity. The problem is…
What it does
When a demand storm hits an LLM fleet, one routing rule decides who still gets the strong model. A playable figure from
In the maker’s words, at launch
OP here: this project was born out of the frustration/paranoia that AI providers are throttling their models when their server load is too high. So, I set out to model and study the problem mathematically to understand what was happening, what I found was quite surprising. The idea seems natural: as the data center demand increases momentarily through the day, throttling their models (either using quantized versions, reducing the context window or lowering the tier of the model to a smaller one) seems appealing as the replacement model in principle uses less electricity. The problem is that this can cause the opposite effect: as users are trying to solve a question, if the degraded AI model gives a bad answer, the user is likely to keep re-asking. On the AI provider side this looks paradoxical: throttling to a lower model creates in fact more demand for their data center. This problem is even worse for agentic workflows, as these are more likely to create a re-ask storm, and maybe explains the outages and anecdotal experiences of users that feel the models are degraded. The model: I used mainly queueing theory arguments solving the optimal scheduling serving for an AI fleet with heterogeneous users solving a finite horizon Dynamic Programming optimization problem. Insights: The industry standard practice of throttling once the number of users in system exceeds a given threshold is in fact what’s causing the problem, the optimal rule implies separating users that won’t feel degradation as much with users that are very sensitive to it (agents and power users vs users doing simple tasks). Limitations: The visualization and paper examples are a toy example to illustrate the problem, only the providers have enough data to properly calibrate these instances. In the paper there are some interesting calibrated instances. Technical Details: The visualization is around 100 lines of flask plus js frontend (LLM assisted with ground truth based on the original numerical example of the paper). Paper with proofs/theory: https://arxiv.org/abs/2608.23986
Does the same job
all alternatives →- WPWe post-trained a model that pen tests instead of refusingJun 2026 · argusred.com · ▲93
Anthropic and OpenAI's publicly available models are explicitly guard-railed so that they refuse offensive tasks. And their cyber-focussed models are gated for enterprises. This leaves SMEs and mid market open to major vulnerabilities. AI can be used as both an adversarial and defensive tool in the world of cyber. A worst case outcome is if only the adversaries have access. Meanwhile, most existing AI cyber tools are just wrappers. The problem is that they still have all the guardrails on from the foundation model where they will inherit its refusals. For this project we've post-trained a…
- IRI ran a language model on a PS2Mar 2026 · github.com · ▲46
The Emotion Engine has 32 MB of RAM total, so the trick is streaming weights from CD-ROM one matrix at a time during the forward pass — only activations, KV cache and embeddings live in RAM. This means models bigger than the RAM can still run, they just read more from disc. Had to build a custom quantized format (PSNT), hack endianness, write a tokenizer pipeline, and most of the PS2 SDK from scratch (releasing that separately). The model itself is also custom — a 10M param Llama-style architecture I trained specifically for this. And it works. On real hardware.
- S1Serve 100 Large AI models on a single GPU with low impact to TTFTNov 2025 · github.com · ▲7
I wanted to build an inference provider for proprietary AI models, but I did not have a huge GPU farm. I started experimenting with Serverless AI inference, but found out that coldstarts were huge. I went deep into the research and put together an engine that loads large models from SSD to VRAM up to ten times faster than alternatives. It works with vLLM, and transformers, and more coming soon. With this project you can hot-swap entire large models (32B) on demand. Its great for: Serverless AI Inference Robotics On Prem deployments Local Agents And Its open source. Let me know if anyone…
- TCThe Cascade Graph – An interactive map of AI and energy constraintsJun 2026 · atomprophet.io · ▲26
Hello, I wanted to share with you all a interactive map of the economics and physics constraints of the AI buildout. It has macro drivers, industrial chokepoints, and where that shows up in markets. I've added 393 nodes and 562 edges to capture other supply / physics constraints as well. There's no sign up, and no pay wall, it's all free. Please let me know what you think!
- IBI built a website showing the likelihood of the AI bubble to popJul 2026 · laurentiugabriel.github.io · ▲11
I built a web page that aggregates data about data center buildup, sovereign fund investments into AI and bottlenecks. The objective is to predict AI race cooldown by looking at a potential decrease of activity involving these elements. The website looks at the quarterly forms from the 5 biggest hyperscalers and adds their CapEx into the mix, calculating a composite index in the end showing how likely it is for the AI race to slow down. Enjoy!
- ABA business SIM where humans beat GPT-5 by 9.8 XNov 2025 · ▲23
Hi HN, Can current AI systems actually run a business? There’s a growing belief that LLM agents can already manage entire teams, replace the entire software stack or even act as an AI CEO. So we built a controlled, measurable environment to evaluate this premise. Why did we build this benchmark? A modern enterprise operates in a dynamic environment with high uncertainty and incomplete information. The CEO has to deal with delayed consequences, staffing/resource tradeoffs and death by a thousand cuts of failure modes. If we ever want AI systems that can meaningfully make operational or…
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 27d ago · cactuscompute.com
Launched alongside, August 2026
the whole month →- TL
Life & fun · 10d ago · louisabraham.github.io
- SA
Hello HN! I found that picking out plausible but diverse skin tones for my digital art and game development projects was kind of difficult, and I got curious about if there was a way to define a color space that made it easy. I've built a color picker and procedural generation algorithm based on the space as well as a bunch of other fun js features and demos throughout the page that use the equations. If you find it interesting, I have lots of explanations of how I built it and what properties the space has. The methodology might be a bit shaky, but hopefully the result is as helpful for…
Life & fun · Aug 2026 · toneyalexander.github.io
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com