Alternatives
Products that do what Run any Llama model finetune and more, instantly does
Hi there, looking for feedback on my new project "Featherless.AI" The idea is to allow users to run all the models on hugging face instantly. Via the OpenAI API compatible endpoint. Why? Because its a real chore to download models and spin up GPUs, especially if you want to test multiple models. Not to mention GPUs cost multiple dollars an hour to rent. And if we want more people to use open source AI, we got to make it easier for them to try and play with all of them. So what if instead of spinning up dedicated GPUs per model (which is what every provider is doing) We can startup a LLM…
- 1
- 28F
Hi HN! I'm just sharing a project I've been working on during the LLM Efficiency Challenge - you can now finetune Llama with QLoRA 5x faster than Huggingface's original implementation on your own local GPU. Some highlights: 1. Manual autograd engine - hand derived backprop steps. 2. QLoRA / LoRA 80% faster, 50% less memory. 3. All kernels written in OpenAI's Triton language. 4. 0% loss in accuracy - no approximation methods - all exact. 5. No change of hardware necessary. Supports NVIDIA GPUs since 2018+. CUDA 7.5+. 6. Flash Attention support via Xformers. 7. Supports 4bit and 16bit…
2023 · github.com
- 3IV
The video demo runs a 7b Model on a normal gaming GPU. I think it already works quite well (accounting for the limited hardware power). :)
2024 · github.com
- 4WM
Try it out! https://glhf.chat/ Hey HN! We’ve been working for the past few months on a website to let you easily run (almost) any open-source LLM on autoscaling GPU clusters. It’s free for now while we figure out how to price it, but we expect to be cheaper than most GPU offerings since we can run the models multi-tenant. Unlike Together AI, Fireworks, etc, we’ll run any model that the open-source vLLM project supports: we don’t have a hardcoded list. If you want a specific model or finetune, you don’t have to ask us for it: you can just paste the Hugging Face link in and…
2024 · glhf.chat
- 5IB
I spent the last few days building out a nicer ChatGPT-like interface to use Mistral 7B and Llama 3 fully within a browser (no deps and installs). I’ve used the WebLLM project by MLC AI for a while to interact with LLMs in the browser when handling sensitive data but I found their UI quite lacking for serious use so I built a much better interface around WebLLM. I’ve been using it as a therapist and coach. And it’s wonderful knowing that my personal information never leaves my local computer. Should work on Desktop with Chrome or Edge. Other browsers are adding WebGPU support as well - see…
2024 · github.com
- 6
- 7LA
A simple mobile web app inspired by Fuzzy-Search/realtime-bakllava that uses llama.cpp server backend with multimodal mode to describe and narrate what the phone camera sees. I built this thing in a few hours using a single ChatGPT thread to generate most things for me and iterate on this project. Here's the workflow: https://chat.openai.com/share/ea84ec69-5617-45e8-8772-ac2dcf...
2023 · github.com
- 8RL
Hello Hacker News! We're Yangqing, Xiang and JJ from lepton.ai. We are building a platform to run any AI models as easy as writing local code, and to get your favorite models in minutes. It's like container for AI, but without the hassle of actually building a docker image. We built and contributed to some of the world's most popular AI software - PyTorch 1.0, ONNX, Caffe, etcd, Kubernetes, etc. We also managed hundreds of thousands of computers in our previous jobs. And we found that the AI software stack is usually unnecessarily complex - and we want to change that. Imagine if you are a…
2023 · lepton.ai
- 9IM
I’m Hayden, a 13-year-old developer based in Australia, and I’ve built a CoT logical thinking and reasoning AI model similar to OpenAI o1. It's powered by open source small models like Llama 3.1 and 3.2 and I would love for you to try it and share your feedback with me. You can try it here: https://ai.pixelverse.tech/app/cortexchat I built it just for fun and launched it a day after the o1 release. It's not perfect yet but its still amazing to see how a detailed prompt can have such a difference on the quality of the LLM response! Please let me know any feedback or…
2024 · ai.pixelverse.tech
- 10AT
I recently built a small open-source tool to benchmark different LLM API endpoints — including OpenAI, Claude, and self-hosted models (like llama.cpp). It runs a configurable number of test requests and reports two key metrics: • First-token latency (ms): How long it takes for the first token to appear • Output speed (tokens/sec): Overall output fluency Demo: https://llmapitest.com/ Code: https://github.com/qjr87/llm-api-test The goal is to provide a simple, visual, and reproducible way to evaluate performance across different LLM providers, including…
2025 · llmapitest.com
- 11IR
Democratisation of local AI is key. I've been working on pushing the limits of commercial hardware, squeezing any extra bit possible. My Scientific Agentic AI hareness helped me to reallocate every single bit of it. I rewrote the Kernel, I went down the CUDA rabbit hole until I have been able to explain any bit and any ms of computational power involved in the process pushing the Qwen 30B-A3B from 8 tok7s to 19 tok/s with llama.cpp up to 22.2 tok/s with my project and 109 tok/s on not novel content and speeding up the prefill by 5-9X
Jul 2026 · github.com
- 12IB
Hey HN, I built a website where you can train Llama 3.1 8b & 70b (4bit) on your data. I use unsloth in the backend and the training is done on H100s which I rent programmatically from Runpod. I'd love some feedback. If you would be interested in using it feel free to book a chat with me: cal.com/hamada/tunellama-intro Happy to give you free credits :) P.S. I'm also looking for a co-founder as I have big plans for this.
2024 · tunellama.com
- 13IB
Hi HN I built a fun little tool: It uses Groq’s LLaMA 3.3 + Puppeteer to analyze a website Then it roasts the design/content/UX with humor And finishes with 3–5 genuinely helpful improvement tips You can try it here: https://ai-roast-vert.vercel.app I wanted to: Practice fast idea-to-launch cycle (built in 2 days) Experiment with a viral-friendly product Monetize with a $0.55 pro version that gives a detailed roast + download Would love your feedback — on the idea, the tone, the usefulness — anything! Thanks in advance
Sep 2025 · ai-roast-vert.vercel.app
- 145L
We've built InferX, a specialized runtime environment that fundamentally changes how LLMs are served. The core problem we solve is the latency bottleneck in AI inference, especially with large models. Current systems waste resources or suffer from painfully slow cold starts. InferX's AI-native architecture, with its "snapshot" technology, enables: * *Sub-2s cold starts:* Spin up models instantly. * *High density:* Serve more LLMs on the same GPUs. * *Optimal efficiency:* Maximize GPU utilization. This isn't just another API; it's a new execution layer designed from the ground up for the…
2025 · github.com
- 15HA
Demo starts at 50m into the video. This was a bit terrifying to record because 2am the previous night everything was totally broken after a major refactor (so that we could add external LLM support as well as local GPUs). But pressure can be a useful force :-D We start with a stack deployed on my laptop without a GPU, pointing to together.ai so we can run open source LLMs easily without having to have access to a GPU. We show simple inference through the ChatGPT-like web interface (with users, sessions etc) and then simple drag'n'drop RAG. Then we show some helix apps defined as yaml: Marvin…
2024 · youtube.com
- 16IB
Hey HN! I just released a suite of AI models for deployment on UAV and other "overhead" devices to provide some understanding of the world below. The objective is to empower all sorts of open-source use cases around search and rescue, wildfire prevention, ground risk mitigation for flight over populated areas etc... The neural networks are trained for a bunch of different devices from big GPUs to tiny edge AI cameras like the Luxonis OAK, with some optimised ones for Nvidia TensorRT and other cool bits and pieces too. The main release package also includes some boilerplate code for running…
2023 · github.com
- 17
- 18S1
I wanted to build an inference provider for proprietary AI models, but I did not have a huge GPU farm. I started experimenting with Serverless AI inference, but found out that coldstarts were huge. I went deep into the research and put together an engine that loads large models from SSD to VRAM up to ten times faster than alternatives. It works with vLLM, and transformers, and more coming soon. With this project you can hot-swap entire large models (32B) on demand. Its great for: Serverless AI Inference Robotics On Prem deployments Local Agents And Its open source. Let me know if anyone…
Nov 2025 · github.com
- 19IB
I had 14,000 photos sitting on a drive and wanted an excuse to play with local vision models and Elixir/Phoenix. I originally tried to get LLaVA to tell me if a photo was 'good' or matched my style, but quickly learned that LLMs have terrible taste. I ended up demoting the LLM to just extract metadata, and built a custom CLIP/Ridge Regression pipeline to actually learn my preferences based on how I rate things. The stack is Phoenix/Oban on the orchestrator side, and Python/FastAPI/Instructor for the AI workers. Happy to answer any questions about the architecture,…
Apr 2026 · qwelian.com
- 20R5
Hi HN, I built OpenGraviton, an open-source AI inference engine that pushes the limits of running extremely large LLMs on consumer hardware. By combining 1.58-bit ternary quantization, dynamic sparsity with Top-K pruning and MoE routing, and mmap-based layer streaming, OpenGraviton can run models far larger than your system RAM—even on a Mac Mini. Early benchmarks: TinyLlama-1.1B drops from ~2GB (FP16) to ~0.24GB with ternary quantization. At 140B scale, models that normally require ~280GB fit within ~35GB packed. Optimized for Apple Silicon with Metal + C++ tensor unpacking, plus…
Mar 2026 · github.com
- 21LT
Hi HN! I'm the author of mere.run a local first inference runtime built around an installable CLI. I believe that whenever possible we should use the stuff we already own (like our Mac laptops, decent machines gathering dust, our gaming PC) and the limited electrical power we have easy access to, like the socket in the wall next to most of us. We shouldn't have to send our data to the cloud hoping some T&C will prevent it from being used in a way that we'd regret. Most of the local AI solutions are technical, involved, and land a curious body in some package hell. People are optimizing for…
Jul 2026 · github.com
- 22IT
I trained the 65b model on my texts so I can talk to myself. It's pretty useless as an assistant, and will only do stuff you convince it to, but I guess it's technically uncensored? I'll leave it up for a bit if you want to chat with it. I posted this to Reddit and had several hundred people talking to it. Salient points from that discussion: LLAMA 1 65b Rank 128 5 epochs Batch size 1, 256 cutoff Trained in the Oobabooga suite using bitsandbytes 4-bit quantization for the lora Loss around 1.5 seems to give the most coherent results Trained on raw text dumps that is then parsed by a crappy…
2023 · airic.serveo.net
- 23AA
Hey HN, I'm an AI enthusiast and I am launching apps that use ML to solve problems that we all have. I realized that there are a lot of deepfake faces on youtube and social media, so I thought it would be useful (and fun!) to have a tool that can bust those AI faces. So I built DeeFace for 2 reasons: 1. Check if the face you're looking at is real or not 2. For fun! It's something that I from half a year ago would look at and aspire to build someday. I think it's soemthing anyone who's just starting out with ML would appreciate. I hope this tool is fun for you to use as it was for me to…
2024 · deefaces.com
- 24WO
Hey guys, I am think of building an open-source version of Perplexity to let devs play around with it. But with all the existing tools available what features would you want? Anything specific? What is missing? Currently working on - 1. Streaming text 2. Citations sources 3. Image and file upload 4. Chat history and storage 5. Temperature and custom instructions If you are in marketing or growth can anyone help me with what to focus on while building such an app? Also here is a very first version. Probably will break and most of the buttons also don’t work, built it in 3 days using Bing and…
2024 · omniplex.vercel.app
Ranked by how close each launch is in meaning, then by votes. Refine with a description →