DuckDB for Kafka Stream Processing
Hello Everyone! We built SQLFlow as a lightweight stream processing engine. We leverage DuckDB as the stream processing engine, which gives SQLFlow the ability to process 10's of thousands of messages a second using ~250MiB of memory! DuckDB also supports a rich ecosystem of sinks and connectors! https://sql-flow.com/docs/category/tutorials/ https://github.com/turbolytics/sql-flow We were tired of running JVM's for simple stream processing, and also of bespoke one off stream processors I would love your feedback, criticisms and/or…
DuckDB for Kafka Stream Processing launched on December 8, 2025 with 77 votes, #384 of 11,628 launches that month and more than 95% of that year's launches.
In plain words
SQLFlow is a lightweight stream processing engine for Kafka that uses DuckDB to process tens of thousands of messages per second while consuming approximately 250MB of memory. It is designed for developers who need stream processing capabilities without running resource-heavy Java virtual machines. The tool offers a rich ecosystem of sinks and connectors for flexible data pipeline integration.
written from the facts on this page · September 2026
From the sources
Quickstart. sqlflow documentation.
Create a stream processor that reads data from Kafka in less than 5 minutes . Run a stream processor that reads a Kafka topic, executes SQL, and writes the output to the console . git clone https://github.com/turbolytics/sql-flow.git && cd sql-flow The turbolytics/sql-flow docker image. Use v1.0.5 or later; this page pulls v1.0.6 . docker pull turbolytics/sql-flow:v1.0.6 Kafka running on your local machine docker compose -f dev/kafka-single.yml up -d The repo’s Python dependencies, for the sample-data script that publishes events into Kafka. sqlflow itself is a Go binary and needs no Python. Dependencies are locked with uv : uv sync Test the sqlflow configuration file sqlflow’s…from sql-flow.com
In the maker’s words, at launch
Hello Everyone! We built SQLFlow as a lightweight stream processing engine. We leverage DuckDB as the stream processing engine, which gives SQLFlow the ability to process 10's of thousands of messages a second using ~250MiB of memory! DuckDB also supports a rich ecosystem of sinks and connectors!…
Does a similar job
all alternatives →
SQLFlow: SQL Stream Processing Using DuckDB2023 · github.com · ▲7I'm a couple years late to the party but DuckDB is blowing my mind. I couldn't find an good embedded DuckDB stream processing solution so I hacked my own, calling it SQLFlow: https://github.com/turbolytics/sql-flow SQLFlow enables writing stream transformations in pure sql, executed using DuckDB. The goal was to create a lightweight, performant stream processing engine using pure SQL transformations, DuckDB didn't disappoint! I'd love your feedback, feature requests, impressions, or just comments. I'd love to turn this into a stable, usable project that people are…
Riko – A Python stream processing engine modeled after Yahoo! Pipes2016 · github.com · ▲283A Python stream processing engine modeled after Yahoo! Pipes - nerevu/riko
Jet – in-memory, fault-tolerant, distributed stream processing2020 · github.com · ▲129Distributed Stream and Batch Processing. Contribute to hazelcast/hazelcast-jet development by creating an account on GitHub.
Denormalized – Embeddable Stream Processing in Rust and DataFusion2024 · github.com · ▲125tl;dr we built an embeddable stream processing engine in Rust using apache DataFusion, check us out at https://github.com/probably-nothing-labs/denormalized Hey HN, We’d like to showcase a very early version of our embeddable stream processing engine called Denormalized. The rise of DuckDB has abundantly made it clear that even for many workloads of Terabyte scale, a single node system outshines the distributed query engines of previous generation such as Spark, Snowflake etc in terms of both performance and cost. Now a lot of workloads DuckDB is used for were normally…
Execute SQL against Bluesky firehose2024 · github.com · ▲100DuckDB for streaming data. Contribute to turbolytics/sql-flow development by creating an account on GitHub.
More dev tools this month
the category →
open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.
Dev tools · 27d ago · github.com
Contribute to reladraw/reladraw development by creating an account on GitHub.
Dev tools · 25d ago · github.com
OpenBot▲362Grok Bot alternative: free, local, open-source, multiplayer
Dev tools · 22d ago · openbot.run
Talanoa▲340Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…
Dev tools · 20d ago · talanoa.email
CoreSpeed▲332One MCP for everything your agents need: apps, memory, tools
Dev tools · 27d ago · corespeed.io
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, December 2025
the whole month →- GP
Life & fun · Dec 2025 · dosaygo-studio.github.io
Hi everyone! My name's Luke and I made the original Jmail here alongside Riley Walz. We had a ton of friends collaborate on building out more of the app suite last night in lieue of DOJ's "Epstein files" release. Please AMA!
Life & fun · Dec 2025 · jmail.world
PlanEat AI▲733AI turns your health goals into a 7-day menu & grocery list
AI · Dec 2025 · planeatai.com
Community, All the HN belong to you. This is an archive of hacker news that fits in your browser. When I made HN Made of Primes I realized I could probably do this offline sqlite/wasm thing with the whole GBs of archive. The whole dataset. So I tried it, and this is it. Have Hacker News on your device. Go to this repo (https://github.com/DOSAYGO-STUDIO/HackerBook): you can download it. Big Query -> ETL -> npx serve docs - that's it. 20 years of HN arguments and beauty, can be yours forever. So they'll never die. Ever. It's the unkillable static archive of HN and it's…
Dev tools · Dec 2025 · hackerbook.dosaygo.com
