Dux, distributed DuckDB-backed dataframes on the Beam
Hey all! I wrote Explorer[1] a good few years ago now with the dream of fast dataframes with a dplyr-like API in a really powerful, ergonomic language (Elixir). It's proved pretty successful. Explorer is used in production at my company, and it's my go-to for quick data analysis. But maintaining it became a true albatross. Polars is an amazing project, but the development process is fast and a lot is very focused on the Python lib. We found that trying to maintain Explorer against Polars was a maintenance nightmare and eventually hit points where we had to give up features and found it…
What it does
In the maker’s words, at launch
Hey all! I wrote Explorer[1] a good few years ago now with the dream of fast dataframes with a dplyr-like API in a really powerful, ergonomic language (Elixir). It's proved pretty successful. Explorer is used in production at my company, and it's my go-to for quick data analysis. But maintaining it became a true albatross. Polars is an amazing project, but the development process is fast and a lot is very focused on the Python lib. We found that trying to maintain Explorer against Polars was a maintenance nightmare and eventually hit points where we had to give up features and found it extremely difficult to update to the latest. We also tried distributing Explorer and only got so far. A reasonable alternative to Spark was always what I wanted, and I could (tantalisingly, frustratingly) see the pieces there in dataframes and the BEAM, but couldn't make it happen. We also always knew that the right direction was to be 'lazy by default', accumulating ops and only executing when the dataframe needs to be realised. But this was very difficult with Polars's Series API and eager/lazy split. Enter DuckDB. A few weeks ago, I made a duckdb backend for Explorer. But in doing so I saw that DuckDB would allow us to realise the lazy-by-default and distributed vision. So I went for it. And here we are. Dux as in ducks as in multiple ducks. Plus an 'x' in the name because, you know, it's Elixir. It's faster than Explorer on a single node. It has a simple, dataframe only API. It distributes arbitrarily on Erlang clusters on the BEAM. Startup is faster than Spark and for many use cases it's simpler and faster. DuckDB functions are all transparently available, as are custom SQL macros. We have a full graph API, as in GraphX/NetworkX. You can install and use any duckdb extensions, including in distribution. And on the maintenance side, it doesn't use a NIF (it depends on the ADBC library[2] and a DuckDB driver) -- the API is primarily about compiling to SQL. DuckDB is incredible for OLAP on out of memory data. Distribution enables fast exploration of in-memory data and real-time applications. The BEAM gives us battle-hardened distribution almost for free. Give it a shot! I'd love feedback and of course PRs are welcome. Oh, I also made a webpage for it[3]. [1] https://github.com/elixir-explorer/explorer [2] https://github.com/livebook-dev/adbc [3] https://dux.now
Does the same job
all alternatives →- VOVaex - Out of Core Dataframes for Python and Fast Visualization2018 · medium.com · ▲126
- DIDataframes in Elixir Backed by Rust2021 · github.com · ▲16
- HPHawk – Pandas-like data analysis for JSON/YAML/CSV in CLI2025 · github.com · ▲5
I built this because I was tired of scrolling through hundreds of lines of AWS CLI JSON output just to find instance status. hawk brings pandas-like operations (select, group_by, aggregations) to CLI with unified syntax across JSON/YAML/CSV. Key features: - Instant data structure overview with `| info` - Same query syntax for all formats - Built in Rust for speed and single-binary distribution Would love feedback from the community!
- DRDuckDB + React = SQLRooms Framework for Analytics Apps Without Backend2025 · sqlrooms.org · ▲9
We’ve built SQLRooms, an open-source framework for creating single-node data analytics apps powered by DuckDB. It lets you build fully client-side, data-centric apps using React and DuckDB running in the browser (via WebAssembly) or in Electron. No server or backend is required — apps can work offline, preserve data privacy, and run queries on large datasets with sub-second performance. Features: - Query large datasets in browser with DuckDB (WASM) - Modular design for building composable data UIs (query editors, dashboards, notebooks, etc.) - Data privacy-preserving AI assistant that can…
- IMI made this tool for navigating pandas datasets2020 · github.com · ▲20
- UIUSearch Images demo in 200 lines of Python2023 · usearch-images.com · ▲9
Hey everyone! I am excited to share updates on four of my & my teams' open-source projects that take large-scale search systems to the next level: USearch, UForm, UCall, and StringZilla. These projects are designed to work seamlessly together, end-to-end—covering everything from indexing and AI to storage and networking. And yeah, they're optimized for x86 AVX2/512 and Arm NEON/SVE hardware. USearch [1]: Think of it as Meta FAISS on steroids. It's now quicker, supports clustering of any granularity, and offers multi-index lookups. Plus, it's got more native bindings than probably…
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 1d ago · opentrailpaper.com
Launched alongside, March 2026
the whole month →
Switch from ChatGPT to Claude with import memory feature
AI · Mar 2026 · claude.com
