Dlt – Python library to automate the creation of datasets

In plain words

Dlt is an open-source Python library that automates the creation of structured datasets from messy, unstructured data sources. It handles data movement from various origins into SQL databases, vector stores, data lakes, cloud storage, and local engines like DuckDB. Designed for Python developers, dlt eliminates repetitive data engineering tasks and removes the need to write custom scripts for each data source integration.

written from the facts on this page · September 2026

From the sources

In the maker’s words, at launch

Hi HN, We're Anna, Adrian, Marcin and Matt, developers of dlt. dlt is an open source library to automatically create datasets out of messy, unstructured data sources. You can use the library to move data from about anywhere into most of well known SQL and vector stores, data lakes, storage buckets, or local engines…

• 
  Automatically extract data from APIs with dlt and OpenAPI2024 · colab.research.google.com · ▲7

  Hi Show HN, we are Dave, Marcin, Alena, and Adrian, authors of data load tool (dlt), a Python library that automatically creates datasets from any kind of messy, unstructured data. We launched dlt on HN 7 months ago with a mission to make getting datasets fast and easy. Now dlt helps its users to code around a thousand new data sources each month and to maintain many thousands of live datasets in production. Today we are releasing *dlt-init-openapi,* a Python CLI tool that generates a dlt data pipeline from any OpenAPI spec. It brings the time to create a dataset down to a few minutes.…

• DA
  dstack – an open-source tool to build data applications easily2020 · ▲134

  Dear HN, I am Riwaj, the cofounder of dstack.ai (https://github.com/dstackai). A few months ago, we built an online service that allows users to publish data visualizations from Python or R. The idea was to build a tool that did not require additional programming or front-end development for publishing data visualizations. Such a code can be invoked from either Jupyter notebook, RMarkdown, Python, or R scripts. Once the data is pushed, it can be accessed via a browser. Open-sourcing dstack: During our customer discovery phase, we realized that dstack.ai should integrate a lot…

• 
  We open sourced our entire text-to-SQL product2024 · github.com · ▲464

  Long story short: We (Dataherald) just open-sourced our entire codebase, including the core engine, the clients that interact with it and the backend application layer for authentication and RBAC. You can now use the full solution to build text-to-SQL into your product. The Problem: modern LLMs write syntactically correct SQL, but they struggle with real-world relational data. This is because real world data and schema is messy, natural language can often be ambiguous and LLMs are not trained on your specific dataset. Solution: The core NL-to-SQL engine in Dataherald is an LLM based agent…

• 
  I wrote a RDBMS (SQLite clone) from scratch in pure Python2023 · github.com · ▲371

  I wrote a relational database management system (RDBMS) (sqlite clone) from scratch in pure Python.

• 
  I built an open-source data pipeline tool in Go2024 · github.com · ▲200

  Every data pipeline job I had to tackle required quite a few components to set up: - One tool to ingest data - Another one to transform it - If you wanted to run Python, set up an orchestrator - If you need to check the data, a data quality tool Let alone this being hard to set up and taking time, it is also pretty high-maintenance. I had to do a lot of infra work, and while this being billable hours for me I didn’t enjoy the work at all. For some parts of it, there were nice solutions like dbt, but in the end for an end-to-end workflow, it didn’t work. That’s why I decided to build an…

• 
  Orchest2020 · orchest.io · ▲112

  An open source tool for creating data science pipelines

• 
  /dev/null▲489

  a social network for devs

  Dev tools · 3d ago · dev-null.sh

• 
  Whiteboard (YC W26) – an open-source IDE for thoughtful software design▲423

  open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.

  Dev tools · 28d ago · github.com

• 
  Reladraw – A diagram language where you decide where to place things▲411

  Contribute to reladraw/reladraw development by creating an account on GitHub.

  Dev tools · 26d ago · github.com

• 
  OpenBot▲362

  Grok Bot alternative: free, local, open-source, multiplayer

  Dev tools · 23d ago · openbot.run

• 
  Talanoa▲346

  Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…

  Dev tools · 21d ago · talanoa.email

• 
  CoreSpeed▲332

  One MCP for everything your agents need: apps, memory, tools

  Dev tools · 28d ago · corespeed.io

• 
  AI Content Genie▲1,071

  AI autopilot for content creation & marketing

  AI · 2023 · typeface.ai

• 
  Nudge 2.0▲1,051

  In-app experiences to activate, retain, & understand users

  Growth · 2023 · nudgenow.com

• 
  Cal.ai Email Assistant▲918

  World's first open source AI scheduling assistant

  AI · 2023 · cal.com

• 
  Design Studio by Tiledesk▲867

  Unlock AI magic for elevated customer engagement, fast

  AI · 2023 · tiledesk.com

• 
  TalkNotes▲854

  Turn your thoughts into actionable notes, fast

  AI · 2023 · talknotes.io

• 
  OpenAPI DevTools – Chrome extension that generates an API spec▲811

  Effortlessly discover API behaviour with a Chrome extension that automatically generates OpenAPI specifications in real time for any app or website.

  Dev tools · 2023 · github.com