Alternatives
Products that do what I built a tool showing how AI providers (should) throttle their models does
OP here: this project was born out of the frustration/paranoia that AI providers are throttling their models when their server load is too high. So, I set out to model and study the problem mathematically to understand what was happening, what I found was quite surprising. The idea seems natural: as the data center demand increases momentarily through the day, throttling their models (either using quantized versions, reducing the context window or lowering the tier of the model to a smaller one) seems appealing as the replacement model in principle uses less electricity. The problem is…
- 1WP
Anthropic and OpenAI's publicly available models are explicitly guard-railed so that they refuse offensive tasks. And their cyber-focussed models are gated for enterprises. This leaves SMEs and mid market open to major vulnerabilities. AI can be used as both an adversarial and defensive tool in the world of cyber. A worst case outcome is if only the adversaries have access. Meanwhile, most existing AI cyber tools are just wrappers. The problem is that they still have all the guardrails on from the foundation model where they will inherit its refusals. For this project we've post-trained a…
Jun 2026 · argusred.com
- 2
- 3IR
The Emotion Engine has 32 MB of RAM total, so the trick is streaming weights from CD-ROM one matrix at a time during the forward pass — only activations, KV cache and embeddings live in RAM. This means models bigger than the RAM can still run, they just read more from disc. Had to build a custom quantized format (PSNT), hack endianness, write a tokenizer pipeline, and most of the PS2 SDK from scratch (releasing that separately). The model itself is also custom — a 10M param Llama-style architecture I trained specifically for this. And it works. On real hardware.
Mar 2026 · github.com
- 4S1
I wanted to build an inference provider for proprietary AI models, but I did not have a huge GPU farm. I started experimenting with Serverless AI inference, but found out that coldstarts were huge. I went deep into the research and put together an engine that loads large models from SSD to VRAM up to ten times faster than alternatives. It works with vLLM, and transformers, and more coming soon. With this project you can hot-swap entire large models (32B) on demand. Its great for: Serverless AI Inference Robotics On Prem deployments Local Agents And Its open source. Let me know if anyone…
Nov 2025 · github.com
- 5NT
Hello HackerNews! I’m excited to share what we’ve been working on at nCompass Technologies: an AI inference* platform that gives you a scalable and reliable API to access any open-source AI model — with no rate limits. We don't have rate limits as optimizations we made to our AI model serving software enable us to support a high number of concurrent requests without degrading quality of service for you as a user. If you’re thinking, well aren’t there a bunch of these already? So were we when we started nCompass. When using other APIs, we found that they weren’t reliable enough to be able to…
2024 · ncompass.tech
- 6TC
Hello, I wanted to share with you all a interactive map of the economics and physics constraints of the AI buildout. It has macro drivers, industrial chokepoints, and where that shows up in markets. I've added 393 nodes and 562 edges to capture other supply / physics constraints as well. There's no sign up, and no pay wall, it's all free. Please let me know what you think!
Jun 2026 · atomprophet.io
- 7AU
I run 3mins.news (https://3mins.news), an AI news aggregator built entirely on Cloudflare Workers. The backend has 10+ cron triggers running every few minutes: RSS fetching, article clustering, LLM calls, email delivery. The problem: Workers Paid Plan has hard monthly limits (10M requests, 1M KV writes, 1M queue ops, etc.). There's no built-in "pause when you hit the limit", CF just starts billing overages. KV writes cost $5/M over the cap, so a retry loop bug can get expensive fast. AWS has Budget Alerts, but those are passive notifications, by the time you read the email,…
Mar 2026
- 8IB
I built a web page that aggregates data about data center buildup, sovereign fund investments into AI and bottlenecks. The objective is to predict AI race cooldown by looking at a potential decrease of activity involving these elements. The website looks at the quarterly forms from the 5 biggest hyperscalers and adds their CapEx into the mix, calculating a composite index in the end showing how likely it is for the AI race to slow down. Enjoy!
Jul 2026 · laurentiugabriel.github.io
- 9AB
Hi HN, Can current AI systems actually run a business? There’s a growing belief that LLM agents can already manage entire teams, replace the entire software stack or even act as an AI CEO. So we built a controlled, measurable environment to evaluate this premise. Why did we build this benchmark? A modern enterprise operates in a dynamic environment with high uncertainty and incomplete information. The CEO has to deal with delayed consequences, staffing/resource tradeoffs and death by a thousand cuts of failure modes. If we ever want AI systems that can meaningfully make operational or…
Nov 2025
- 10C5
Hi HN community! My name is Kane and I'm on the product team at www.usage.ai , a cloud cost optimization company. After honing our product on AWS, I'm excited to announce our availability for customers on GCP and Azure! At a high level, Usage insures committed use-discounts (CUDs) to reduce commitment risk and enable higher savings for companies on the cloud. With traditional CUDs, you commit to a certain level of usage over a specified period in exchange for discounted rates from the cloud provider. However, if your actual usage falls short of the committed amount, you may not fully realize…
2024
- 11
- 12WB
Hey HN: Kaveh here, founder of https://www.usage.ai/ We help companies drive down AWS, GCP, and Azure spend. Why? Because the way it's done now is a pain. DevOps and Software Engineers end up spending time managing costs rather than focusing on business problems. I have been building Usage AI for almost 4 years now (4 year anniversary in 1 month from now!) with an incredible group of founding people. We started as a product just to help lower AWS EC2 costs, and now we do all major AWS services (such as RDS, OpenSearch, ElastiCache, and Redshift with more on the way) and other…
2024
- 13IN
Tl;dr: I trained a classifier to route to the least expensive model and reasoning depth to complete the request. Coupling that with additional automated token efficiency techniques has yielded 3x usage for the same spend. For anyone interested in trying it themselves: https://nerfguard.com Various teammates and I switched over to Codex from Claude Code recently. We still bounce between the tools, but Codex’s speed and steerability coupled with performance gains were hard to ignore. One of the downsides was that the per token pricing kicked in way sooner. This is happening across…
Jun 2026
- 14IB
Hi everyone, I've been working on a side project over quarantine called Usage.ai and I finally feel comfortable enough to launch it. We're a service that plugs directly into AWS, automatically finds savings, and applies those savings at the press of a confirmation button all without ever needing to go to an AWS console. I'd love to get HN's thoughts on it! Demo: https://www.loom.com/share/2a6f1c8e4c214914a1cdd88c6fdec4ac Link: https://www.usage.ai/
2020
- 155L
We've built InferX, a specialized runtime environment that fundamentally changes how LLMs are served. The core problem we solve is the latency bottleneck in AI inference, especially with large models. Current systems waste resources or suffer from painfully slow cold starts. InferX's AI-native architecture, with its "snapshot" technology, enables: * *Sub-2s cold starts:* Spin up models instantly. * *High density:* Serve more LLMs on the same GPUs. * *Optimal efficiency:* Maximize GPU utilization. This isn't just another API; it's a new execution layer designed from the ground up for the…
2025 · github.com
- 16DF
There is an adversarial relationship between developers and big model labs. Model labs charged developers higher API prices to subsidize their own agent harness offerings. Think Anthropic charging 5x higher Claude API prices to subsidize consumer subscriptions. So Cursor in a way was subsidizing their own direct competitor. DeepSeek V4 Flash totally inverted this relationship. Now you have a model that beats even Sonnet in some benchmarks and is totally opensourced. Now inference providers are racing to the bottom to optimize and give cheaper hosting. Every player with a non-SOTA is now…
Jun 2026 · rtrvr.ai
- 17
- 18
- 19IR
Democratisation of local AI is key. I've been working on pushing the limits of commercial hardware, squeezing any extra bit possible. My Scientific Agentic AI hareness helped me to reallocate every single bit of it. I rewrote the Kernel, I went down the CUDA rabbit hole until I have been able to explain any bit and any ms of computational power involved in the process pushing the Qwen 30B-A3B from 8 tok7s to 19 tok/s with llama.cpp up to 22.2 tok/s with my project and 109 tok/s on not novel content and speeding up the prefill by 5-9X
Jul 2026 · github.com
- 20RA
We built RapidFire AI, an open-source Python tool to speed up LLM fine-tuning and post-training with a powerful level of control not found in most tools: Stop, resume, clone-modify and warm-start configs on the fly—so you can branch experiments while they’re running instead of starting from scratch or running one after another. - Works within your OSS stack: PyTorch, HuggingFace TRL/PEFT), MLflow. - Hyperparallel search: launch as many configs as you want together, even on a single GPU - Dynamic real-time control: stop laggards, resume them later to revisit, branch promising configs in…
Sep 2025 · github.com
- 21
- 22FA
Hi HN, We're excited to introduce Fixstars AIBooster, our new performance engineering tool designed to significantly accelerate AI model training while optimizing GPU utilization. AIBooster provides: Real-time monitoring of GPU, CPU, memory, and power consumption. Clear visibility into performance bottlenecks, helping developers optimize AI workloads. Proven acceleration of AI training processes—users commonly achieve up to 2-3x speed improvements. Significant cost savings by maximizing infrastructure efficiency. It's free to try, requires minimal setup, and integrates seamlessly into your…
2025 · fixstars.com
- 23RA
Hi HN folks, I have been building AI agents for quite some time now. The shift has gone from LLM + Tools → LLM Workflows → Agent + Tools + Memory, and now we are finally seeing true agency emerge: agents as systems composed of tools, command-line access, fine-grained system capabilities, and memory. This way of building agents is powerful, and I believe it is here to stay. But the real question is: are the systems powering these agents ready for that future? I do not think so. Using Docker for a single agent is not going to scale well, because agents need to be lightweight and fast. LLMs…
Mar 2026 · github.com
- 24IB
Hi HN, When I was in FAANG, I despised ticketing. I was always the eng that refused to update. Then when I was a tech lead, I had the pain of pinging everyone for updates because they hated it too. Things would go out of date, and then no one would really know if we were on track. We also had little visibility into what the team was doing outside of pinging/standups. So I made an AI that looks at every commit you push and updates tickets for you. It can create tickets if there isn't one, update, leave comments, tag associated commits etc... Basically, it's an "AI Board Babysitter" so…
2025 · usevectra.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →