Sycamore – an LLM-powered semantic data preparation system for search
We’re Aryn and yesterday we came out of stealth: blog.aryn.ai. As part of that, we released Sycamore: https://github.com/aryn-ai/sycamore. Sycamore is an LLM-powered semantic data preparation system for building search applications. It introduces a distributed set-based abstraction, a DocSet, that makes processing a large document collection as easy as reading a single document. Sycamore makes it easy to use LLMs to transform and enrich your unstructured data and prepare it for search. It comes with a scalable distributed runtime, built on Ray, that makes it easy to go…
Sycamore – an LLM-powered semantic data preparation system for search launched on September 29, 2023 with 18 votes, #1,402 of 4,228 launches that month and more than 75% of that year's launches. The site was still online in September 2026, 3 years on.
In plain words
Sycamore is an LLM-powered system for preparing unstructured data to build search applications. It introduces a DocSet abstraction that simplifies processing large document collections, allowing developers to transform and enrich data with a few lines of code. The system includes a distributed runtime built on Ray for scaling from prototypes to production. It's designed for developers building search applications who need to process documents like PDFs, extract entities, compute embeddings, and load results into search platforms.
written from the facts on this page · September 2026
From the sources
🍁 Sycamore is an LLM-powered search and analytics platform for unstructured data. - aryn-ai/sycamore
In the maker’s words, at launch
We’re Aryn and yesterday we came out of stealth: blog.aryn.ai. As part of that, we released Sycamore: https://github.com/aryn-ai/sycamore. Sycamore is an LLM-powered semantic data preparation system for building search applications. It introduces a distributed set-based abstraction, a DocSet, that…
Does a similar job
all alternatives →
Turn any website into a knowledge base for LLMs2024 · embedding.io · ▲305I built this tool because I wanted a way to just take a bunch of URLs or domains, and query their content in RAG applications. It takes away the pain of crawling, extracting content, chunking, vectorizing, and updating periodically. I'm curious to see if it can be useful to others. I meant to launch this six months ago but life got in the way...
PDF to MD by LLMs – Extract Text/Tables/Image Descriptives by GPT4o2024 · github.com · ▲191I've developed a Python API service that uses GPT-4o for OCR on PDFs. It features parallel processing and batch handling for improved performance. Not only does it convert PDF to markdown, but it also describes the images within the PDF using captions like `[Image: This picture shows 4 people waving]`. In testing with NASA's Apollo 17 flight documents, it successfully converted complex, multi-oriented pages into well-structured Markdown. The project is open-source and available on GitHub. Feedback is welcome.
Data Bonsai: a Python package to clean your data with LLMs2024 · github.com · ▲47I've been doing some data cleaning for my fine tuning projects using LLMs, and decided to just build a package for it as a side project. Check it out here: https://github.com/databonsai/databonsai Some features: - categorization (labelling), transformation and decomposition (text into structured format) - validates llm outputs - batch mode batches up the inputs/outputs so you don't send the prompt (schema, fewshot examples) for every row of data, saving a significant amount of tokens There are some similarities to the Instructor repo, but this is simpler and made for…
Get your unstructured data AI-ready in minutes2024 · tonic.ai · ▲38Hey HN! We're excited to announce the launch of Tonic Textual, the secure data lakehouse for LLMs. Simply stated, Tonic Textual allows you to build generative AI systems on your own unstructured data without having to spend time extracting and standardizing your data. In minutes you can build automated, scalable unstructured data pipelines that extract, centralize, standardize, and enrich data from your documents into an AI-optimized format ready for embedding, fine-tuning, and ingesting into a vector database. While in-flight, we also scan for sensitive information and protect it via…
I just open sourced my document/website extractor for Vision-LLMs2024 · github.com · ▲37Hi HackerNews, Lately, I have seen an explosion in posts offering paid APIs/services to get unstructured data into LLMs (i.e. langchain extract, ragflow, unstructured, unstract, just to name a few) and I have been largely disappointed by them, either because they fail to implement multimodal support, fail to give good context for "really tricky" PDFs / Word docs / Powerpoints, or are just plain difficult to use. In light of all these posts I figured I'd share my solution that has been working smoothly for me and my clients. I put it up on GitHub for free so you can check it…
folioApr 2026 · usefolio.ai · ▲74hi HN! a couple of months ago I had to analyze a few thousand audio recordings to help identify issues with customer support. i was able to get some raw high-level initial results with python scripts invoking LLM APIs, but they were too general and unhelpful. writing basic prompts is easy, but tuning them and making them specific enough to ensure no faint signal is missed is hard. you need to iterate through the data with an initial prompt, segment the data into different buckets, chain another prompt for each bucket etc. Then you need to constantly review the raw data to tweak the prompts…
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 21d ago · stillwet.art
FastRouter.ai▲364Route requests to the right LLM for cost, latency & quality
AI · 23d ago · fastrouter.ai
Source: Show HN post ↗ · launch text quoted in part
