ColiVara – State of the Art RAG API with Vision Models
we have been working on ColiVara and wanted to show it to the community. ColiVara is an api-first implementation of the ColPali paper using ColQwen2 as the LLM model. It works exactly like RAG from the end-user standpoint - but using vision models instead of chunking and text-processing for documents. Why should anyone working with RAG care? ColPali makes information retrieval from visual document types - like PDFs - better. Colivara is a suite of services that allows you to store, search, and retrieve documents based on their visual embedding built on top of ColPali. (We are not affiliated…
In plain words
ColiVara is an API-first retrieval system that uses vision models instead of traditional text chunking to search and retrieve documents like PDFs. Built on the ColPali paper and ColQwen2 model, it works as a RAG alternative by storing and retrieving documents based on visual embeddings rather than text processing. This approach handles complex document layouts containing text, images, tables, and various formatting elements. ColiVara is designed for developers working with document retrieval who need better handling of visually complex files.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
we have been working on ColiVara and wanted to show it to the community. ColiVara is an api-first implementation of the ColPali paper using ColQwen2 as the LLM model. It works exactly like RAG from the end-user standpoint - but using vision models instead of chunking and text-processing for documents. Why should anyone working with RAG care? ColPali makes information retrieval from visual document types - like PDFs - better. Colivara is a suite of services that allows you to store, search, and retrieve documents based on their visual embedding built on top of ColPali. (We are not affiliated with the ColPali team in anyway, although we are big fans of their work!) Information retrieval from PDFs is hard because they contain various components: Text, images, tables, different headings, captions, complex layouts, etc. For this, parsing PDFs currently requires multiple complex steps: 1. OCR 2. Layout recognition 3. Figure captioning 4. Chunking 5. Embedding Not only are these steps complex and time-consuming, but they are also prone to error. This is where ColPali comes into play. But what is ColPali? ColPali combines: • Col -> the contextualized late interaction mechanism introduced in ColBERT • Pali -> with a Vision Language Model (VLM), in this case, PaliGemma (note - both us and the ColPali team moved from PaliGemma to use Qwen models) And how does it work? During indexing, the complex PDF parsing steps are replaced by using "screenshots" of the PDF pages directly. These screenshots are then embedded with the VLM. At inference time, the query is embedded and matched with a late interaction mechanism to retrieve the most similar document pages. Ok - so what exactly ColiVara does? ColiVara is an API (with a Python SDK) that makes this whole process easy and viable for production workloads. With 1-line of code - you get a SOTA retrieval in your RAG system. We optimized how the embeddings are stored (using pgVector and halfvecs) as well as re-implemented the scoring to happen in Postgres, similar to and building on pgVector work with Cosine Similarity. All what the user have to do is: 1. Upsert a document to ColiVara to index it 2. At query time - perform a search and get the top-k pages We support advanced filtering based on arbitrary document and collection metadata as well. So, we support re-ranking use cases and hybrid search. State of the art? We started this whole journey when we tried to do RAG over clinical trials and medical literature. We simply had too many failures and up to 30% of the paper was lost or malformed. This is just not our experience, in the ColPali paper - on average ColPali outperformed Unstructured + BM25 + captioning by 15+ points. ColiVara with its optimizations is is 20+ points. We used NCDG@5 - which is similar to Recall but more demanding, as it measure not just if the right results are returned, but if they returned in the correct order. You can see our full eval results here: https://github.com/tjmlabs/ColiVara-eval If this sounds like something you could use, check it out on GitHub: https://github.com/tjmlabs/ColiVara It’s fair-source with an FSL license (similar to Sentry), and we’d love to hear how you’d use it or any feedback you might have. Additionally - our eval repo is public and we continuously run against major releases. You are welcome to run the evals independently: https://github.com/tjmlabs/ColiVara-eval
Does a similar job
all alternatives →- FBFastGraphRAG – Better RAG using good old PageRank2024 · github.com · ▲457
Hey there HN! We’re Antonio, Luca, and Yuhang, and we’re excited to introduce Fast GraphRAG, an open-source RAG approach that leverages knowledge graphs and the 25 years old PageRank for better information retrieval and reasoning. Building a good RAG pipeline these days takes a lot of manual optimizations. Most engineers intuitively start from naive RAG: throw everything in a vector database and hope that semantic search is powerful enough. This can work for use cases where accuracy isn’t too important and hallucinations are tolerable, but it doesn’t work for more difficult queries that…
- OSOpen-source Rule-based PDF parser for RAG2024 · github.com · ▲293
The PDF parser is a rule based parser which uses text co-ordinates (boundary box), graphics and font data. The PDF parser works off text layer and also offers a OCR option to automatically use OCR if there are scanned pages in your PDFs. The OCR feature is based off a modified version of tika which uses tesseract underneath. The PDF Parser offers the following features: * Sections and subsections along with their levels. * Paragraphs - combines lines. * Links between sections and paragraphs. * Tables along with the section the tables are found in. * Lists and nested lists. * Join content…
- HWHow we leapfrogged traditional vector based RAG with a 'language map'2024 · twitter.com · ▲162
TL;DR: Vector-based RAG performs poorly for many real-world applications like codebase chats, and you should consider 'language maps'. Part of our mission at Mutable.ai is to make it much easier for developers to build and understand software. One of the natural ways to do this is to create a codebase chat, that answer questions about your repo and help you build features. It might seem simple to plug in your codebase into a state-of-the-art LLM, but LLMs have two limitations that make human-level assistance with code difficult: 1. They currently have context windows that are too small to…
- COCognita – open-source RAG framework for modular applications2024 · github.com · ▲142
Hey HN, exciting news! Our RAG framework, Cognita (https://github.com/truefoundry/cognita), born from collaborations with diverse enterprises, is now open-source. Currently, it offers seamless integrations with Qdrant and SingleStore. In recent weeks, numerous engineers have explored Cognita, providing invaluable insights and feedback. We deeply appreciate your input and encourage ongoing dialogue (share your thoughts in the comments – let's keep this ‘open source’). While RAG is undoubtedly powerful, the process of building a functional application with it can feel…
- GAGibRAM an in-memory ephemeral GraphRAG runtime for retrievalJan 2026 · github.com · ▲60
Hi HN, I have been working with regulation-heavy documents lately, and one thing kept bothering me. Flat RAG pipelines often fail to retrieve related articles together, even when they are clearly connected through references, definitions, or clauses. After trying several RAG setups, I subjectively felt that GraphRAG was a better mental model for this kind of data. The Microsoft GraphRAG paper and reference implementation were helpful starting points. However, in practice, I found one recurring friction point: graph storage and vector indexing are usually handled by separate systems, which…
- PVPageIndex – Vectorless RAG2025 · github.com · ▲192
Not all improvements come from adding complexity — sometimes it's about removing it. PageIndex takes a different approach to RAG. Instead of relying on vector databases or artificial chunking, it builds a hierarchical tree structure from documents and uses reasoning-based tree search to locate the most relevant sections. This mirrors how humans approach reading: navigating through sections and context rather than matching embeddings. As a result, the retrieval feels transparent, structured, and explainable. It moves RAG away from approximate "semantic vibes" and toward explicit reasoning…
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 18d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 19d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 27d ago · cactuscompute.com
Launched alongside, November 2024
the whole month →
- IB
I wasn't quite sure if this qualified as "Show HN" given you can't really download it and try it out. However, dang said[0]: > If it's hardware or something that's not so easy to try out over the internet, find a different way to show how it actually works—a video, for example, or a detailed post with photos. Hopefully I did that? Additionally, I've put code and a detailed guide for the netboot computer management setup on GitHub: https://github.com/kentonv/lanparty Anyway, if this shouldn't have been Show HN, I apologize! [0]…
Dev tools · 2024 · lanparty.house
