Run any Llama model finetune and more, instantly
Hi there, looking for feedback on my new project "Featherless.AI" The idea is to allow users to run all the models on hugging face instantly. Via the OpenAI API compatible endpoint. Why? Because its a real chore to download models and spin up GPUs, especially if you want to test multiple models. Not to mention GPUs cost multiple dollars an hour to rent. And if we want more people to use open source AI, we got to make it easier for them to try and play with all of them. So what if instead of spinning up dedicated GPUs per model (which is what every provider is doing) We can startup a LLM…
What it does
In the maker’s words, at launch
Hi there, looking for feedback on my new project "Featherless.AI" The idea is to allow users to run all the models on hugging face instantly. Via the OpenAI API compatible endpoint. Why? Because its a real chore to download models and spin up GPUs, especially if you want to test multiple models. Not to mention GPUs cost multiple dollars an hour to rent. And if we want more people to use open source AI, we got to make it easier for them to try and play with all of them. So what if instead of spinning up dedicated GPUs per model (which is what every provider is doing) We can startup a LLM server, which multiple users can share, and instantly hot swap it in seconds the model its serving??? Make sense right, thats what we do for apps, or shared CPU PHP apps. --- Turns out thats an incredibly complex problem, that even Hugging Face, together AI and various companies had failed to successfully build. Turns out swapping out GB's of data from disk to GPU in subseconds is hard. And involves optimizing every small thing in the chain - had to setup and tune a high speed storage cluster - find a GPU provider with crazy networking speed, and would allow you to scale up/down - custom write a new pipeline for streaming data from high speed storage cluster - custom write code to stream that data, straight into GPU as fast as possible After all that, you now have a GPU server, that can serve one model at a time, but can switch models quickly. So to make this idea economical (since im not billing anyone $8/hour) - you will need to setup a cluster, to handle multiple request concurrently - while writing routing code, to ensure all the request for the same models go to the same server - unless it hits the limit, then you need to start load balancing between servers assigned to that model - and to backoff the load balancing, so that you can free up servers - to be hot swapped for other request Oh also downloading 450+ models, apparently takes up tons of TB's, of expensive high speed clustered storage. But the end result, a highly dynamic scaling (up or down) infrastructure, to the exact cluster workload, for a large collection of HF models (more models then GPUs of course). All so that people can use "all the open source AI models", for a few dollars a month. While not worrying about token pricing. So do give it a try, the free trial account can test any 8B model, and a subscribe account has full OpenAI API access. And give us some feedback, and maybe even a product hunt vote! PS: This is a work in progress, I yet to rewrite the custom pipeline code for non-llama / non-rwkv models, thats why we are starting with only those 2 architectures first. But we do plan to scale to ALL public models on huggingface. Also downloading the other 1000+ llama models does take a long time.
Does the same job
all alternatives →
- IMI'm 13 and built an OpenAI o1-like AI model – would love your feedback2024 · ai.pixelverse.tech · ▲6
I’m Hayden, a 13-year-old developer based in Australia, and I’ve built a CoT logical thinking and reasoning AI model similar to OpenAI o1. It's powered by open source small models like Llama 3.1 and 3.2 and I would love for you to try it and share your feedback with me. You can try it here: https://ai.pixelverse.tech/app/cortexchat I built it just for fun and launched it a day after the o1 release. It's not perfect yet but its still amazing to see how a detailed prompt can have such a difference on the quality of the LLM response! Please let me know any feedback or…
- IRI run 30B 22tok/s, 109tok/s not novel,6GB/16GB RAM overcoming llama.cppJul 2026 · github.com · ▲5
Democratisation of local AI is key. I've been working on pushing the limits of commercial hardware, squeezing any extra bit possible. My Scientific Agentic AI hareness helped me to reallocate every single bit of it. I rewrote the Kernel, I went down the CUDA rabbit hole until I have been able to explain any bit and any ms of computational power involved in the process pushing the Qwen 30B-A3B from 8 tok7s to 19 tok/s with llama.cpp up to 22.2 tok/s with my project and 109 tok/s on not novel content and speeding up the prefill by 5-9X
- IBI built a website where you can easily fine-tune Llama 3.1 models2024 · tunellama.com · ▲8
Hey HN, I built a website where you can train Llama 3.1 8b & 70b (4bit) on your data. I use unsloth in the backend and the training is done on H100s which I rent programmatically from Runpod. I'd love some feedback. If you would be interested in using it feel free to book a chat with me: cal.com/hamada/tunellama-intro Happy to give you free credits :) P.S. I'm also looking for a co-founder as I have big plans for this.
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 26d ago · cactuscompute.com
Launched alongside, June 2024
the whole month →
La Growth Machine▲1,216Create personalized, multi-channel conversations at scale
AI · 2024 · lagrowthmachine.com
PyjamaHR▲1,132Hiring on autopilot. The AI applicant tracking system (ATS).
Work · 2024 · pyjamahr.com