API to turn entire websites into Markdown
While building mendable - we found that feeding LLMs well-structured markdown improved accuracy. We also found it surprisingly hard. We found some great tools online, but none reliably handled the entire process. We wanted an API that took a URL, crawled the pages in the URL, and gave us an easy-to-use, up-to-date markdown we could feed into our index. So, we released an open-source repo and an API that crawls and turns entire websites into a markdown with just a few lines of code The API handles: - Crawling without consistent sitemaps - Infra to handle running many crawling jobs - Proxying,…
In plain words
Firecrawl is an API that crawls websites and converts them into clean, structured markdown suitable for processing by language models. It handles sites without consistent sitemaps, manages large-scale crawling jobs, and converts content including images and tables. The service includes open-source code and a free API tier with built-in integrations for popular AI frameworks like LangChain and Llama Index.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
While building mendable - we found that feeding LLMs well-structured markdown improved accuracy. We also found it surprisingly hard. We found some great tools online, but none reliably handled the entire process. We wanted an API that took a URL, crawled the pages in the URL, and gave us an easy-to-use, up-to-date markdown we could feed into our index. So, we released an open-source repo and an API that crawls and turns entire websites into a markdown with just a few lines of code The API handles: - Crawling without consistent sitemaps - Infra to handle running many crawling jobs - Proxying, hosting headless browsers at scale - Conversion to clean markdown - Caching - Handling images, videos (soon), and tables(soon) - LLM extraction (soon) It is open source, and we also offer an easy-to-use API that starts free. It has built-in loaders for both @llama_index and @langchain. Excited to see people try it
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 1d ago · opentrailpaper.com
Launched alongside, April 2024
the whole month →
Supabase▲2,328The Postgres developer platform is now generally available
Dev tools · 2024 · supabase.com
Cal.com Platform▲1,602Build your pixel-perfect booking experience with Atoms
Dev tools · 2024 · cal.com
deco.cx 2.0▲1,080Build web apps 10x faster with Deno, JSX, TS & Tailwind
Dev tools · 2024 · decocms.com
