Alternatives
Products that do what LLMKube – Kubernetes for Local LLMs with GPU Acceleration does
Hi HN! I built LLMKube, a Kubernetes operator for deploying GPU-accelerated LLMs in production. One command gets you from zero to inference with full observability. Why this exists: Regulated industries (healthcare, defense, finance) need air-gapped LLM deployments, but existing tools are either single-node only (Ollama) or lack GPU optimization and SLO enforcement. LLMKube bridges the gap. What's working: - 17x speedup with NVIDIA GPUs (64 tok/s on Llama 3.2 3B vs 4.6 tok/s CPU) - One command: llmkube deploy llama-3b --gpu (auto CUDA setup, scheduling, layer offloading) -…
- 1TL
Hey HN, we wanted to share our repo where we fine-tuned Llama 3.1 on Google TPUs. We’re building AI infra to fine-tune and serve LLMs on non-NVIDIA GPUs (TPUs, Trainium, AMD GPUs). The problem: Right now, 90% of LLM workloads run on NVIDIA GPUs, but there are equally powerful and more cost-effective alternatives out there. For example, training and serving Llama 3.1 on Google TPUs is about 30% cheaper than NVIDIA GPUs. But developer tooling for non-NVIDIA chipsets is lacking. We felt this pain ourselves. We initially tried using PyTorch XLA to train Llama 3.1 on TPUs, but it was rough: xla…
2024 · github.com
- 2L3
Hi everyone, I'm kinda involved in some retrogaming and with some experiments I ran into the following question: "It would be possible to run transformer models bypassing the cpu/ram, connecting the gpu to the nvme?" This is the result of that question itself and some weekend vibecoding (it has the linked library repository in the readme as well), it seems to work, even on consumer gpus, it should work better on professional ones tho
Feb 2026 · github.com
- 3
- 4FL
I've been playing around with https://github.com/zphang/minimal-llama/ and https://github.com/tloen/alpaca-lora/blob/main/finetune.py, and wanted to create a simple UI where you can just paste text, tweak the parameters, and finetune the model quickly using a modern GPU. To prepare the data, simply separate your text with two blank lines. There's an inference tab, so you can test how the tuned model behaves. This is my first foray into the world of LLM finetuning, Python, Torch, Transformers, LoRA, PEFT, and Gradio. Enjoy!
2023 · github.com
- 58F
Hi HN! I'm just sharing a project I've been working on during the LLM Efficiency Challenge - you can now finetune Llama with QLoRA 5x faster than Huggingface's original implementation on your own local GPU. Some highlights: 1. Manual autograd engine - hand derived backprop steps. 2. QLoRA / LoRA 80% faster, 50% less memory. 3. All kernels written in OpenAI's Triton language. 4. 0% loss in accuracy - no approximation methods - all exact. 5. No change of hardware necessary. Supports NVIDIA GPUs since 2018+. CUDA 7.5+. 6. Flash Attention support via Xformers. 7. Supports 4bit and 16bit…
2023 · github.com
- 6
- 7GO
Hello HN, we're Philip and Louis from Glasskube (https://github.com/glasskube/glasskube). We're working on an open-source package manager for Kubernetes. It's an alternative to tools like Helm or Kustomize, primarily focused on making deploying, updating, and configuring Kubernetes packages simpler and a lot faster. Here is a demo video (https://www.youtube.com/watch?v=aIeTHGWsG2c#t=17s) with quick start instructions. Most developers working with Kubernetes use Helm, an open-source tool created during a hackathon nine years ago. However, with the rapid…
2024 · github.com
- 8
- 9LF
100% bootstrapped new startup. It lets you fine tune Mistral-7B and SDXL. In particular, for the LLM fine tuning we implemented a dataprep pipeline that turns websites/pdfs/doc files into question-answer pairs for training the small LLM using an big LLM. It includes a GPU scheduler that can do finegrained GPU memory scheduling (Kubernetes can only do whole-GPU, we do it per-GB of GPU memory to pack both inference and fine tuning jobs into the same fleet) to fit model instances into GPU memory to optimally trade off user facing latency with GPU memory utilization It's a pretty…
2023 · docs.helix.ml
- 10OR
Hi HN A few folks and I have been working on this project for a couple weeks now. After previously working on the Docker project for a number of years (both on the container runtime and image registry side), the recent rise in open source language models made us think something similar needed to exist for large language models too. While not exactly the same as running linux containers, running LLMs shares quite a few of the same challenges. There are "base layers" (e.g. models like Llama 2), specific configuration to run correctly (parameters, temperature, context window sizes etc). There's…
2023 · github.com
- 11WM
Try it out! https://glhf.chat/ Hey HN! We’ve been working for the past few months on a website to let you easily run (almost) any open-source LLM on autoscaling GPU clusters. It’s free for now while we figure out how to price it, but we expect to be cheaper than most GPU offerings since we can run the models multi-tenant. Unlike Together AI, Fireworks, etc, we’ll run any model that the open-source vLLM project supports: we don’t have a hardcoded list. If you want a specific model or finetune, you don’t have to ask us for it: you can just paste the Hugging Face link in and…
2024 · glhf.chat
- 12
- 13TV
May 2026 · github.com
- 14
- 15LD
I spent days trying to deploy DeepSeek on a server this year. Install Ubuntu, NVIDIA drivers, CUDA, Docker, configure vLLM, debug memory issues, tune performance settings. Every deployment was different. Every server had its own quirks. Worse still, these issues are more pronounced on non-NVIDIA accelerators, such as Ascend or Intel NPU. So, we made LLMOne, which will automates this. You can use it at bare metal (via BMC) or SSH (coming soon) into an existing server, select models, and it handles everything: OS installation, driver setup, inference engine configuration, model deployment, and…
2025 · github.com
- 16LL
Hey, recently I took inspiration from llama.cpp, ollama, and many other similar tools that enable inference of LLMs locally, and I just finished building a Llama inference engine for the 8B model in CUDA C. I recently wanted to explore my newly founded interest in CUDA programming and my passion for machine learning. This project only makes use of the native CUDA runtime api and cuda_fp16. The inference takes place in fp16, so it requires around 17-18GB of VRAM (~16GB for model params and some more for intermediary caches). It doesn’t use cuBLAS or any similar libraries since I wanted to be…
2025 · github.com
- 17
- 18CR
Clawbernetes turns OpenClaw into an AI-native infrastructure manager. Instead of YAML, Helm charts, and kubectl — you have a conversation. "Deploy Llama 70B on the node with the most VRAM" → agent selects the best node, pulls the image, starts the container with GPU passthrough, sets up health monitoring. "Why is inference slow?" → checks GPU temps, VRAM, CPU load. "GPU 0 at 89°C — thermal throttling. Want me to reduce batch size?" 23 crates, 74K lines of Rust, 1,866 tests, zero unsafe in core. Supports CUDA, Metal, ROCm, Vulkan, and CPU SIMD. Components: - clawnode: node agent with 80+…
Feb 2026 · github.com
- 19RG
Hi HN! We’re Yann, Edouard, and Bastien from Koyeb (https://www.koyeb.com/). We’re building a platform to let you deploy full-stack apps on high-performance hardware around the world, with zero configuration. We provide a “global serverless feeling”, without the hassle of re-writing all your apps or managing k8s complexity [1]. We built Scaleway, a cloud service provider where we designed ARM servers and provided them as cloud servers. During our time there, we saw customers struggle with the same issues while trying to deploy full-stack applications and APIs resiliently. As…
2023 · koyeb.com
- 20LC
Complete llama.cpp tutorial for 2026. Install, compile with CUDA/Metal, run GGUF models, tune all inference flags, use the API server, speculative decoding, and benchmark your hardware. https://vucense.com/dev-corner/llama-cpp-tutorial-run-gguf-m...
Apr 2026
- 21LA
G'day, HN! I'm one of the maintainers of `llm`. I've been working alongside a trusty group of contributors to bring this project to life, and we're now at a point where we're ready to share it with the world. Large language models (LLMs) are taking the computing world by storm due to their emergent abilities that allow them to perform a wide variety of tasks, including translation, summarization, code generation, and even some degree of reasoning. However, the ecosystem around LLMs is still in its infancy, and it can be difficult to get started with these models. `llm` is a one-stop shop for…
2023 · github.com
- 22LG
Jul 2026 · github.com
- 23GV
Hi folks, We worked hard over the last couple of weeks to build a new package manager for Kubernetes that makes it super simple to install packages and solves some of the shortcomings of helm (dependency management, multi namespace installations, a GUI and CLI as first class interfaces). We just released our source code on GitHub: https://github.com/glasskube/glasskube/ and looking for feedback!
2024 · glasskube.dev
- 24OS
Hey HN, I am the founder of Tensorlake. Prototyping LLM applications have become a lot easier, building decision making LLM applications that work on constantly updating data is still very challenging in production settings. The systems engineering problems that we have seen people face are - 1. Reliably process ingested content in real time if the application is sensitive to freshness of information. 2. Being able to bring in any kind of model, and run different parts of the pipeline on GPUs and CPUs. 3. Fault Tolerance to ingestion spike, compute infrastructure failure. 4. Scaling compute,…
2024 · getindexify.ai
Ranked by how close each launch is in meaning, then by votes. Refine with a description →