LLMKube – Kubernetes for Local LLMs with GPU Acceleration
Hi HN! I built LLMKube, a Kubernetes operator for deploying GPU-accelerated LLMs in production. One command gets you from zero to inference with full observability. Why this exists: Regulated industries (healthcare, defense, finance) need air-gapped LLM deployments, but existing tools are either single-node only (Ollama) or lack GPU optimization and SLO enforcement. LLMKube bridges the gap. What's working: - 17x speedup with NVIDIA GPUs (64 tok/s on Llama 3.2 3B vs 4.6 tok/s CPU) - One command: llmkube deploy llama-3b --gpu (auto CUDA setup, scheduling, layer offloading) -…
What it does
In the maker’s words, at launch
Hi HN! I built LLMKube, a Kubernetes operator for deploying GPU-accelerated LLMs in production. One command gets you from zero to inference with full observability. Why this exists: Regulated industries (healthcare, defense, finance) need air-gapped LLM deployments, but existing tools are either single-node only (Ollama) or lack GPU optimization and SLO enforcement. LLMKube bridges the gap. What's working: - 17x speedup with NVIDIA GPUs (64 tok/s on Llama 3.2 3B vs 4.6 tok/s CPU) - One command: llmkube deploy llama-3b --gpu (auto CUDA setup, scheduling, layer offloading) - Production observability: Prometheus + Grafana + DCGM GPU metrics out of the box - OpenAI-compatible API endpoints - Terraform configs for GKE GPU clusters with auto-scale to zero Tech: Kubernetes CRDs, llama.cpp with CUDA, NVIDIA GPU Operator, cost-optimized spot instances (~$50-150/mo dev workloads). Status: v0.2.0 production-ready for single-GPU deployments on standard K8s clusters. Multi-GPU and multi-node model sharding on the roadmap. Apache 2.0 licensed. Would love feedback from anyone running LLMs in production! Website: https://llmkube.com GitHub: https://github.com/Defilan/LLMKube
Does the same job
all alternatives →
- LDLLMOne – Deploy LLMs from bare metal to production in hours2025 · github.com · ▲5
I spent days trying to deploy DeepSeek on a server this year. Install Ubuntu, NVIDIA drivers, CUDA, Docker, configure vLLM, debug memory issues, tune performance settings. Every deployment was different. Every server had its own quirks. Worse still, these issues are more pronounced on non-NVIDIA accelerators, such as Ascend or Intel NPU. So, we made LLMOne, which will automates this. You can use it at bare metal (via BMC) or SSH (coming soon) into an existing server, select models, and it handles everything: OS installation, driver setup, inference engine configuration, model deployment, and…
- CRClawbernetes – Replace kubectl with conversation (Rust)Feb 2026 · github.com · ▲5
Clawbernetes turns OpenClaw into an AI-native infrastructure manager. Instead of YAML, Helm charts, and kubectl — you have a conversation. "Deploy Llama 70B on the node with the most VRAM" → agent selects the best node, pulls the image, starts the container with GPU passthrough, sets up health monitoring. "Why is inference slow?" → checks GPU temps, VRAM, CPU load. "GPU 0 at 89°C — thermal throttling. Want me to reduce batch size?" 23 crates, 74K lines of Rust, 1,866 tests, zero unsafe in core. Supports CUDA, Metal, ROCm, Vulkan, and CPU SIMD. Components: - clawnode: node agent with 80+…
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 27d ago · cactuscompute.com
Launched alongside, November 2025
the whole month →
- IB
Life & fun · Nov 2025 · bitsnpieces.dev
- BBoing▲782
Life & fun · Nov 2025 · boing.greg.technology