Alternatives
Products that do what Airgapped Offline RAG – Run LLMs Locally with Llama, Mistral, & Gemini does
I've built an airgapped Retrieval-Augmented Generation (RAG) system for question-answering on documents, running entirely offline with local inference. Using Llama 3, Mistral, and Gemini, this setup allows secure, private NLP on your own machine. Perfect for researchers, data scientists, and developers who need to process sensitive data without cloud dependencies. Built with Llama C++, LangChain, and Streamlit, it supports quantized models and provides a sleek UI for document processing. Check it out, contribute, or suggest new features!
- 1DA
I've built an advanced RAG (Retrieval-Augmented Generation) pipeline from scratch to demystify the complex mechanics of modern LLM-powered Question Answering systems. This repository features: -- An implementation of a sub-question query engine from scratch to answer complex user questions. -- Illustrative explanations that unveil the inner workings of the system. -- An analysis of the challenges I faced while working with the system, like prompt engineering and cost estimation. -- Qualitative comparison with similar frameworks like LlamaIndex, offering a broader perspective. Key Takeaway:…
2023 · github.com
- 2IB
I spent the last few days building out a nicer ChatGPT-like interface to use Mistral 7B and Llama 3 fully within a browser (no deps and installs). I’ve used the WebLLM project by MLC AI for a while to interact with LLMs in the browser when handling sensitive data but I found their UI quite lacking for serious use so I built a much better interface around WebLLM. I’ve been using it as a therapist and coach. And it’s wonderful knowing that my personal information never leaves my local computer. Should work on Desktop with Chrome or Edge. Other browsers are adding WebGPU support as well - see…
2024 · github.com
- 3IV
The video demo runs a 7b Model on a normal gaming GPU. I think it already works quite well (accounting for the limited hardware power). :)
2024 · github.com
- 4TA
I built this tool because I wanted a way to just take a bunch of URLs or domains, and query their content in RAG applications. It takes away the pain of crawling, extracting content, chunking, vectorizing, and updating periodically. I'm curious to see if it can be useful to others. I meant to launch this six months ago but life got in the way...
2024 · embedding.io
- 5RP
Hey hacker news, We’re the cofounders at Psychic.dev (http://psychic.dev) where we help companies connect LLMs to private data. With the launch of Llama 2, we think it’s finally viable to self-host an internal application that’s on-par with ChatGPT, so we did exactly that and made it an open source project. We also included a vector DB and API server so you can upload files and connect Llama 2 to your own data. The RAG in RAGstack stands for Retrieval Augmented Generation, a technique where the capabilities of a large language model (LLM) are augmented by retrieving information…
2023 · github.com
- 6FL
I've been playing around with https://github.com/zphang/minimal-llama/ and https://github.com/tloen/alpaca-lora/blob/main/finetune.py, and wanted to create a simple UI where you can just paste text, tweak the parameters, and finetune the model quickly using a modern GPU. To prepare the data, simply separate your text with two blank lines. There's an inference tab, so you can test how the tuned model behaves. This is my first foray into the world of LLM finetuning, Python, Torch, Transformers, LoRA, PEFT, and Gradio. Enjoy!
2023 · github.com
- 7
- 8
- 9MI
Hi HN! I lead product at Vectara and we've just released a new LLM in our platform that outperforms GPT4 and Gemini 1.5 Pro on RAG tasks. Vectara is a Retrieval Augmented Generation (RAG) platform primarily deployed as a SaaS service which includes a generous free tier so you can try it for free. The way we've been able to offer a "better but cheaper" is that we focus a lot of our attention on taking smaller models (which can be hosted in a cost efficient way) and fine tuning them to specific tasks: in this case RAG. This ends up with a model that is less capable of arbitrary tasks like…
2024 · vectara.com
- 10TL
Hey HN, we wanted to share our repo where we fine-tuned Llama 3.1 on Google TPUs. We’re building AI infra to fine-tune and serve LLMs on non-NVIDIA GPUs (TPUs, Trainium, AMD GPUs). The problem: Right now, 90% of LLM workloads run on NVIDIA GPUs, but there are equally powerful and more cost-effective alternatives out there. For example, training and serving Llama 3.1 on Google TPUs is about 30% cheaper than NVIDIA GPUs. But developer tooling for non-NVIDIA chipsets is lacking. We felt this pain ourselves. We initially tried using PyTorch XLA to train Llama 3.1 on TPUs, but it was rough: xla…
2024 · github.com
- 11IB
We show the potential of modern, embedded graph databases in the browser by demonstrating a fully in-browser chatbot that can perform Graph RAG using Kuzu (the graph database we're building) and WebLLM, a popular in-browser inference engine for LLMs. The post retrieves from the graph via a Text-to-Cypher pipeline that translates a user question into a Cypher query, and the LLM uses the retrieved results to synthesize a response. As LLMs get better, and WebGPU and Wasm64 become more widely adopted, we expect to be able to do more and more in the browser in combination with LLMs, so a lot of…
2025 · blog.kuzudb.com
- 12IB
hey hn, I built an open-source Perplexity clone that can run local LLMs and cloud LLMs. It's fully self-hostable through Docker and uses ollama to support local LLMs. The demo video in the repository shows me running it locally with llama3 on my M1 Macbook Pro. I'm open to any suggestions or feedback, thanks!
2024 · github.com
- 13FT
I wrote a small local tool to transcribe audio notes (Whisper/Parakeet). Code: https://github.com/bilawalriaz/lazy-notes I wanted to process raw transcripts locally without OpenRouter. Llama 3.2 3B with a prompt was decent but incomplete, so I tried SFT. I fine-tuned Llama 3.2 3B to clean/analyze dictation and emit structured JSON (title, tags, entities, dates, actions). Data: 13 real memos → Kimi K2 gold JSON → ~40k synthetic + gold; keys canonicalized. Chutes.ai (5k req/day). Training: RTX 4090 24GB, ~4h, LoRA (r=128, α=128, dropout=0.05), max seq 2048,…
2025 · bilawal.net
- 14AW
Lately, I've been tinkering with llama.cpp and the ollama server. The speed of these tools caught my attention, even on my modest 4060 setup. I was quite impressed with the generation quality of models like Mistral. But I was a bit unhappy at the same time because whenever I explore a topic, there is a lot of typing involved when using the chat interface. So I needed a tool to not only give a response but also generate a set of "suggestions" which can be explored further just by clicking. My experience in front-end development is limited. Nonetheless, I tinkered together a small web app to…
2023 · github.com
- 15AF
Hello HN! I wanted to share a side project of mine: a fully standalone app that uses a local LLM (specifically Mistral 7B) for document interaction using RAG. It supports a wide variety of document types and is designed to run right after installation. There are still quite a few bugs to iron out and many features I'm excited to add. Considering this is my first time working on such a project, I'm really happy with the progress and am learning a ton! The project is open source, so I please feel free to check out my work and leave any feedback :)
2024 · github.com
- 16IB
Hey HN! Over the past few weeks, I’ve been working on DataBridge, an open-source solution for data ingestion and querying across text, PDFs, images, and videos. In our latest update, we’ve added a fully local deployment option: - No internet required – Runs entirely offline. - Customizable Models – Supports any LLM and embedding model via Ollama (with options for any other private providers) - Extensibility – You can plug in your own models or tools easily. This local-first approach ensures better privacy, security, and flexibility, especially for teams dealing with sensitive data. You can…
2025 · github.com
- 17IB
Hey all! I wanted to share this project I've been working on that can maybe help you or your developer friends out. I built a RAG system for our product a while back and didn't realize how easy they were to get started. So I put together my learnings into this online course. It's not quite ready but if you sign up and mention HackerNews, I can get you early access. I'm looking to get feedback on the following: (1) materials — is it engaging & did you learn something? (2) UI/UX of the platform — did you have any issues that prevented you from starting or finishing the tutorial? (3)…
2024 · takehomes.com
- 18SO
Henry, Matt and James here – we’re building an open source toolkit that makes it easy to integrate an LLM-powered copilot that talks to your API into software products. It works by calling API endpoints which you choose to expose to it. This lets the chatbot complete tasks within your software in response to natural language queries. It’s also open source, so you don’t have to send user data to another 3rd party. We support Llama 2, but we haven’t fine-tuned Llama 2 yet (coming soon) so highest accuracy is seen with GPT-4 or fine-tuned GPT-3.5 (much faster). We started working together 2…
2023 · github.com
- 19RA
Hey HN! If you’re at all interested in LLMs/LLM-apps, you’ve probably heard of RAG: Retrieval-Assisted Generation, i.e. retrieving relevant documents to give to your LLM as context to answer user queries. Today, I’m releasing RAGatouille v0.0.1, whose aim is to make it as easy as can be to improve your RAG pipelines by leveraging state-of-the-art Information Retrieval research. As of right now, there’s quite a big gap between common everyday practice and the IR literature, and a lot of the gap is because there just aren’t good ways to quickly try out and leverage SotA IR techniques.…
2024 · github.com
- 20GA
We’ve just launched Gradient — an API that helps you build private LLMs that you own. We simplify inference and fine-tuning on open-source LLMs such as llama2, and you only pay by the token. Our API platform makes it possible for you to create private models with a single API call. Run inference on your fine tuned model instantly with no cold boot (and no need to pay for compute costs). The product is truly on demand - when you run fine tuning and inference on our platform, there's nearly 0 startup latency for these API calls. And you're not paying for the compute, you just pay for the…
2023 · gradient.ai
- 21PO
We are the developers of Phoenix, which we released in April of this year with a goal of bringing LLM observability to the notebook. In the time since, the growth of LLM frameworks and complex agent workflows led us to add support for LLM spans and traces and introduce a simple Eval harness for testing the data from those spans. The latest Traces & Spans release of Phoenix offers: -Out of the box tracing for LlamaIndex and LangChain -Fully local execution, no data sent anywhere, outside of your own LLM calls -Ability to get a common dataframe format across frameworks back to a notebook for…
2023 · github.com
- 22DL
2024 · github.com
- 23IB
Hey HN, I built a website where you can train Llama 3.1 8b & 70b (4bit) on your data. I use unsloth in the backend and the training is done on H100s which I rent programmatically from Runpod. I'd love some feedback. If you would be interested in using it feel free to book a chat with me: cal.com/hamada/tunellama-intro Happy to give you free credits :) P.S. I'm also looking for a co-founder as I have big plans for this.
2024 · tunellama.com
- 24HA
Demo starts at 50m into the video. This was a bit terrifying to record because 2am the previous night everything was totally broken after a major refactor (so that we could add external LLM support as well as local GPUs). But pressure can be a useful force :-D We start with a stack deployed on my laptop without a GPU, pointing to together.ai so we can run open source LLMs easily without having to have access to a GPU. We show simple inference through the ChatGPT-like web interface (with users, sessions etc) and then simple drag'n'drop RAG. Then we show some helix apps defined as yaml: Marvin…
2024 · youtube.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →