Curator – an open-source library for synthetic data generation
Synthetic data generation is an essential step in training and evaluating LLMs/Agents/RAG pipelines, but tooling around this is still lacking. We're introducing Curator, an open-source library designed to streamline the data curation process. While there are many libraries to prompt LLMs, the semantics of generating synthetic data is different from prompting. For example, we need to process a large number of prompts (sometimes in millions or more) while accepting some failures, utilize several stages of prompting, incorporate human feedback, and filter out bad data using verifiers…
Curator – an open-source library for synthetic data generation launched on January 15, 2025 with 13 votes, #1,099 of 6,142 launches that month and more than 75% of that year's launches. The site was still online in September 2026, 1 year on.
In plain words
Curator is an open-source library for generating synthetic data to train and evaluate large language models, agents, and RAG pipelines. It handles large-scale data generation across multiple API providers and local models, recovers from failures, and incorporates human feedback and data filtering through verifiers. The library uses structured outputs to build complex generation pipelines and enables visualization of curated datasets.
written from the facts on this page · September 2026
From the sources
Synthetic data curation for post-training and structured data extraction - bespokelabsai/curator
In the maker’s words, at launch
Synthetic data generation is an essential step in training and evaluating LLMs/Agents/RAG pipelines, but tooling around this is still lacking. We're introducing Curator, an open-source library designed to streamline the data curation process. While there are many libraries to prompt LLMs, the semantics of…
Does a similar job
all alternatives →
Open-source AI data generator (now hosted)2025 · metabase.com · ▲39Hey HN! A few months ago we shared our AI dataset generator as an open source repo, and the response was incredible (https://news.ycombinator.com/item?id=44388093). We got requests from folks who wanted to use it without the hosting overhead, so we created both options: a hosted version (https://www.metabase.com/ai-data-generator for instant use and the source code fully open (https://github.com/metabase/dataset-generator) for anyone who wants to self-host or contribute. Looking forward to seeing how you use it and what you build on top of…
Demystifying Advanced RAG Pipelines2023 · github.com · ▲131I've built an advanced RAG (Retrieval-Augmented Generation) pipeline from scratch to demystify the complex mechanics of modern LLM-powered Question Answering systems. This repository features: -- An implementation of a sub-question query engine from scratch to answer complex user questions. -- Illustrative explanations that unveil the inner workings of the system. -- An analysis of the challenges I faced while working with the system, like prompt engineering and cost estimation. -- Qualitative comparison with similar frameworks like LlamaIndex, offering a broader perspective. Key Takeaway:…
- ATA tool to seed your dev database with real data2022 · ▲129
A bunch of developers and myself have created RepliByte - an open-source tool to seed a development database from a production database. Features: - Support data backup and restore for PostgreSQL, MySQL and MongoDB - Replace sensitive data with fake data - Works on large database (> 10GB) (read Design) - Database Subsetting: Scale down a production database to a more reasonable size - Start a local database with the prod data in a single command - On-the-fly data (de)compression (Zlib) - On-the-fly data de/encryption (AES-256) - Fully stateless (no server, no daemon) and lightweight…
Open-source model and scorecard for measuring hallucinations in LLMs2023 · vectara.com · ▲65Hi all! This morning, we released a new Apache 2.0 licensed model on HuggingFace for detecting hallucinations in retrieval augmented generation (RAG) systems. What we've found is that even when given a "simple" instruction like "summarize the following news article," every LLM that's available hallucinates to some extent, making up details that never existed in the source article -- and some of them quite a bit. As a RAG provider and proponents of ethical AI, we want to see LLMs get better at this. We've published an open source model, a blog more thoroughly describing our methodology (and…
Improving Synthetic Data Distributions with Conditional Sampling2022 · tonic.ai · ▲17Read expert insights on synthetic data, data privacy, and modern software testing. Explore articles from Tonic.ai’s AI engineering team.
React-like Declarative DSL for building synthetic LLM datasetsNov 2025 · github.com · ▲10We were frustrated with dataset generation DX: ad‑hoc scripts and JSON templates break as flows branch, tool‑calls drift, and reproducibility is hard. So we built Torque: a schema‑first, declarative, fully typesafe DSL to generate conversational datasets. What it is: Declarative DSL - Compose conversations like React components Fully Typesafe - Zod schemas with complete type inference Provider Agnostic - Generate with any AI SDK provider (OpenAI, Anthropic, DeepSeek, vLLM, LLaMA.cpp etc.) AI-Powered Content - Generate realistic varied datasets automatically without complicated scripts Faker…
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 20d ago · stillwet.art
PixVerse R2▲381A real-time world model you can explore and change
AI · 29d ago · world.pixverse.video
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, January 2025
the whole month →
Hello! I'm Byran. I spent the past ~6 months engineering a laptop from scratch. It's fully open-source on GH at: https://github.com/Hello9999901/laptop
Dev tools · 2025 · byran.ee
- TITetris in a PDF▲1,289
I realized that the PDF engines of modern desktop browsers (PDFium and PDF.js) support JavaScript with enough I/O primitives to make a basic game like Tetris. It was a bit tricky to find a union of features that work in both engines, but in the end it turns out that showing/hiding annotation "fields" works well to make monochrome pixels, and keyboard input can be achieved by typing in a text input box. All in all it's quite janky but a nice reminder of how general purpose PDF scripting can be. The linked PDF is all ASCII so you can just open it in a text editor, or have a look at…
Life & fun · 2025 · th0mas.nl
21st.dev▲1,211I originally built this app to organize and reuse interesting React + Tailwind components I came across on Twitter, inspired by the shadcn/ui approach. Later, a friend started contributing their own components, and it got me thinking—this could grow into something bigger, like an NPM for design engineers to share and use components. Would love to hear your thoughts!
Dev tools · 2025 · 21st.dev
Create lifelike, personalized AI avatars from text prompts
AI · 2025 · jogg.ai
CSS library based on Counter Strike 1.6 UI.
Life & fun · 2025 · cs16.samke.me