Turn any website into a knowledge base for LLMs
I built this tool because I wanted a way to just take a bunch of URLs or domains, and query their content in RAG applications. It takes away the pain of crawling, extracting content, chunking, vectorizing, and updating periodically. I'm curious to see if it can be useful to others. I meant to launch this six months ago but life got in the way...
Turn any website into a knowledge base for LLMs launched on July 30, 2024 with 305 votes, #167 of 5,184 launches that month and more than 95% of that year's launches. The site was still online in September 2026, 2 years on.
In plain words
This tool converts websites into searchable knowledge bases for large language models used in RAG applications. It automates the entire process of crawling URLs or domains, extracting content, chunking text, creating vector embeddings, and maintaining updates over time. The tool is designed for developers and teams building AI applications who need to integrate external website content without manual data preparation work.
written from the facts on this page · September 2026
From the sources
Turn any website into a knowledge base for LLMs with Embedding.io. Easily crawl, chunk, and vectorize web content for seamless integration with large language models. Start for free and explore our powerful API for efficient data collection and querying.
Crawl, chunk, and vectorize any website so you can use their content with LLMs. Use our API or web interface to make a collection. This holds the pages or websites you want to use. Add web pages to your collection. We will handle the rest, including updates. Once your collection is ready, query it using our API. We keep the data updated for you. A credit is a unit of measurement used to track your usage of our services. Each page you add to your collection consumes one credit. Additionally, each time a page is updated, it also consumes one credit. You can embed any publicly available content on the internet, including websites, blogs, documentation, and more. No, your website does not need…from embedding.io
Does a similar job
all alternatives →
Demystifying Advanced RAG Pipelines2023 · github.com · ▲131I've built an advanced RAG (Retrieval-Augmented Generation) pipeline from scratch to demystify the complex mechanics of modern LLM-powered Question Answering systems. This repository features: -- An implementation of a sub-question query engine from scratch to answer complex user questions. -- Illustrative explanations that unveil the inner workings of the system. -- An analysis of the challenges I faced while working with the system, like prompt engineering and cost estimation. -- Qualitative comparison with similar frameworks like LlamaIndex, offering a broader perspective. Key Takeaway:…
- IBI built a browser-based tutorial for learning RAG for LLMs2024 · takehomes.com · ▲25
Hey all! I wanted to share this project I've been working on that can maybe help you or your developer friends out. I built a RAG system for our product a while back and didn't realize how easy they were to get started. So I put together my learnings into this online course. It's not quite ready but if you sign up and mention HackerNews, I can get you early access. I'm looking to get feedback on the following: (1) materials — is it engaging & did you learn something? (2) UI/UX of the platform — did you have any issues that prevented you from starting or finishing the tutorial? (3)…
I wrote an open-source browser alternative for Computer Use for any LLM2024 · github.com · ▲180Hey HN, I made Browser-Use, an open-source tool that lets (all Langchain supported) LLMs execute tasks directly in the browser just with function calling. It allows you to build agents that interact with web elements using natural language prompts. We created a layer that simplifies website interaction for LLMs by extracting xPaths and interactive elements like buttons and input fields (and other fancy things). This enables you to design custom web automation and scraping functions without manual inspection through DevTools. Hasn't this been done a lot of times? Good question, as a general…
How we leapfrogged traditional vector based RAG with a 'language map'2024 · twitter.com · ▲162TL;DR: Vector-based RAG performs poorly for many real-world applications like codebase chats, and you should consider 'language maps'. Part of our mission at Mutable.ai is to make it much easier for developers to build and understand software. One of the natural ways to do this is to create a codebase chat, that answer questions about your repo and help you build features. It might seem simple to plug in your codebase into a state-of-the-art LLM, but LLMs have two limitations that make human-level assistance with code difficult: 1. They currently have context windows that are too small to…
- IBIn-Browser Graph RAG with Kuzu-WASM and WebLLM2025 · blog.kuzudb.com · ▲158
We show the potential of modern, embedded graph databases in the browser by demonstrating a fully in-browser chatbot that can perform Graph RAG using Kuzu (the graph database we're building) and WebLLM, a popular in-browser inference engine for LLMs. The post retrieves from the graph via a Text-to-Cypher pipeline that translates a user question into a Cypher query, and the LLM uses the retrieved results to synthesize a response. As LLMs get better, and WebGPU and Wasm64 become more widely adopted, we expect to be able to do more and more in the browser in combination with LLMs, so a lot of…
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 19d ago · stillwet.art
Source: Show HN post ↗
Launched alongside, July 2024
the whole month →
Many years ago, I made VJ softwares (to mix live visuals in clubs) for unexpected platforms like the Game Boy Advance, the Playstation 2 and the Raspberry Pi. This year, I’m back with a new web-app: Pikimov. Inspired by Photopea (a free Photoshop clone), I created this web-based motion design & video editor as an alternative to After Effects, to fill empty void. It's free, without signup, without cloud uploads (your files stay on your machine), and your projects are not used for AI models training.
AI · 2024 · pikimov.com
