Pantry – Brew/Docker for LLMs
Started this a few months ago because I wanted a better test runner for an entirely different LLM-based project, then got completely nerdsniped by making LLMs as easy as possible to hotswap into other projects. It's still on the early side but it's finally at the point where I would happily use it for my original project, so I figured I'd post it. It's also—almost by accident—fully remote compatible (permissioning system included!), so you can host it on a box and then connect a program to it remotely.
Does the same job
all alternatives →- IBI built an open-source answer engine that runs local LLMs2024 · github.com · ▲5
hey hn, I built an open-source Perplexity clone that can run local LLMs and cloud LLMs. It's fully self-hostable through Docker and uses ollama to support local LLMs. The demo video in the repository shows me running it locally with llama3 on my M1 Macbook Pro. I'm open to any suggestions or feedback, thanks!
- PUPlock: Use a local LLM from anywhere in your OS2024 · github.com · ▲6
After seeing a cool demo of a hack on Twitter, I built a cross platform version of it that works well and uses streaming. From anywhere on Mac and Linux, trigger Ollama and optionally feed it your clipboard. I built it yesterday and it's already very useful to me. I'm pretty excited about it and wanted to share!
- ULUse local LLMs to organize your files2025 · github.com · ▲6
Hi everyone, Just wanted to share a use case where local LLMs are genuinely helpful for daily workflows: file organization. I've been working on a C++ desktop app called AI File Sorter – it uses local LLMs via `llama.cpp` to help organize messy folders like `Downloads` or `Desktop`. Not sort files into folders solely based on extension or filename patterns, but based on what each file actually is supposed to do or does. Basically: what would normally take me a great deal of time for dragging and sorting can now be done in a few. It's cross-platform (Windows/macOS/Linux), and fully…
- LVLaunch VM workloads securely and instantaneously, without VMs2021 · ▲12
Hello HN! We've been working on a new hypervisor https://kwarantine.xyz that can run strongly isolated containers. This is still a WIP, but we wanted to give the community an idea about our approach, its benefits, and various use cases it unlocks. Today, VMs are used to host containers, and make up for the lack of strong security as well as kernel isolation in containers. This work adds this missing security piece in containers. We plan on launching a free private beta soon. Meanwhile, we'd deeply appreciate any feedback, and happy to answer any questions here or on our slack…
- KPKV-psi, using Linux PSI to to trim an LLM KV cacheJun 2026 · github.com · ▲8
I thought it'd be interesting to use Linux PSI (Pressure Stall Information) for an LLM runtime to trim the KV cache. This is mainly useful imo for edge devices like the Jetson Orin super nano kit which have unified memory. I haven't benched much, but plan to do so more over time and see if I can make a real use of it as I run local LLMs. Let me know if it makes sense :P (I of course vibed this idea)
- OSOpen-source LLM for data labeling2024 · huggingface.co · ▲6
And you can try out the models live here: https://labs.refuel.ai/playground
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 26d ago · cactuscompute.com
