Pruna AI – Inference Optimization Engine
Hello Hacker News! I am Bertrand from Pruna AI. With my associates, John, Rayan, and Stephan, we are fellow researchers in AI efficiency and reliability coming from TUM. We are building an optimization engine that combines compression methods (e.g. quantization, pruning, compilation, batching…) in the aim of saving compute power when running AI models. This optimization engine take one base model as input and returns a compressed model as output. It aims to help for two things: - Make various AI models faster and/or smaller for various hardware (because they can require significant…
What it does
In the maker’s words, at launch
Hello Hacker News! I am Bertrand from Pruna AI. With my associates, John, Rayan, and Stephan, we are fellow researchers in AI efficiency and reliability coming from TUM. We are building an optimization engine that combines compression methods (e.g. quantization, pruning, compilation, batching…) in the aim of saving compute power when running AI models. This optimization engine take one base model as input and returns a compressed model as output. It aims to help for two things: - Make various AI models faster and/or smaller for various hardware (because they can require significant compute power to run). - Easily apply one, but also, multiple compression methods on AI models (because it can take a lot of development time to compress models for production). Currently, Pruna is designed only for inference optimization, not yet training. It focuses on Pytorch models and runs on Linux. It can be deployed in Docker, and is available either self-hosted via CLI (https://docs.pruna.ai/en/latest/setup/pip.html) or via the AWS Marketplace (https://aws.amazon.com/marketplace/pp/prodview-nqi4r52e2qnry). For the last year, to ensure that our product was good enough, we did multiple things: - We built a workflow to automatically scrape various Hugging Face models, run them through our tool, and push back the compressed version to Hugging Face (see 7,500 models available on Hugging Face (https://huggingface.co/PrunaAI). - We also created a benchmark page (special for Flux, soon for Llama) to showcase the results of all our internal testing: Flux Pruna Benchmark(https://flux-pruna-benchmark.vercel.app/). Every company we meet asks, “Do you have numbers?”—and this isn’t just a feature, it’s our way of being transparent about what we bring to the table. - We’ve prepared examples loaded in Google Colabs and documentation to explain what the compression methods do (https://docs.pruna.ai/en/latest/index.html). In terms of compression methods, we aimed to integrate both existing and new compression methods that lead to efficiency gains. We are naturally interested if you have suggestions for other ones. On the backend, we’ve implemented a token system (https://docs.pruna.ai/en/latest/setup/token.html). The token serves as a unique identifier when using the package. Upon your first call to the smash function https://docs.pruna.ai/en/latest/user_manual/smash.html), your token is automatically generated and printed in the console. FYI, for now, we’ve adopted a freemium model (up to 100 hours of runtime per month) with a soft limit (you can exceed it, theoretically, to avoid downtime – we’ll see how it goes if there’s abuse) as we’re still evaluating the best monetization strategy. Our end goal is to combine open-source with feature-gating for enterprises, but we’re not quite there yet. Think of this as an intermediate step. I’m really happy we get to share this with you all. Thanks for reading! Please let us know your thoughts and questions in the comments.
Does the same job
all alternatives →
- S1Serve 100 Large AI models on a single GPU with low impact to TTFTNov 2025 · github.com · ▲7
I wanted to build an inference provider for proprietary AI models, but I did not have a huge GPU farm. I started experimenting with Serverless AI inference, but found out that coldstarts were huge. I went deep into the research and put together an engine that loads large models from SSD to VRAM up to ten times faster than alternatives. It works with vLLM, and transformers, and more coming soon. With this project you can hot-swap entire large models (32B) on demand. Its great for: Serverless AI Inference Robotics On Prem deployments Local Agents And Its open source. Let me know if anyone…
- IBI built an open-source AI system for drones2023 · github.com · ▲5
Hey HN! I just released a suite of AI models for deployment on UAV and other "overhead" devices to provide some understanding of the world below. The objective is to empower all sorts of open-source use cases around search and rescue, wildfire prevention, ground risk mitigation for flight over populated areas etc... The neural networks are trained for a bunch of different devices from big GPUs to tiny edge AI cameras like the Luxonis OAK, with some optimised ones for Nvidia TensorRT and other cool bits and pieces too. The main release package also includes some boilerplate code for running…
- WBWe built a powerful and free AI research assistant2024 · join.modularmind.app · ▲7
Hi everyone, We have been developing a platform to enable professionals to build AI assistants to help them through their work. After a few months, we realized people are trying to sell basic functionalities that can be built from scratch in a couple of hours. Due to this, individuals who are not familiar with the current SOTA are misinformed about the potential of generative models. So, we decided to open up some of our most popular templates as standalone tools for free to empower individuals and set a solid standard for what people should expect. We believe the barrier to accessing…
- MAMoondream, a small vision language model that runs on 8GB of RAM2023 · github.com · ▲9
I've been working on training this small vision language model for the last month - excited to release the first prototype today! It is based on SigLIP (image encoder), Phi-1.5 (text model) and trained using the LLaVa-1.5 training dataset. It runs reasonably fast on CPU with ~8GB of RAM in full 32-bit precision. There's plenty of room to speed it up and reduce memory consumption by quantizing the model. I posted a video of it running on my M2 Macbook Air (on CPU not MPS, so performance should be comparable on other hardware) on Twitter to demonstrate inference speed:…
- ATA tool that turns everyday computers into your own AI cloud2024 · github.com · ▲5
I have a favour to ask. I’ve been working for a while on Kalavai, a project to make distributed AI easy. There are brilliant tools out there to help AI hobbyists and devs on the software layer (shout out to vLLM and llamacpp amongst many others!) but it’s a jungle out there when it comes to procuring and managing the necessary hardware resources and orchestrating them. This has always led me to compromise on the size of the models I end up using (quantized versions, smaller models) to save cost or to play within the limits of my rig. Today I am happy to share the first public version of our…
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 16d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 26d ago · cactuscompute.com
