Dev tools · alternatives · 2026
24 alternatives to DuckDB for Kafka Stream Processing
Hello Everyone! We built SQLFlow as a lightweight stream processing engine. We leverage DuckDB as the stream processing engine, which gives SQLFlow the ability to process 10's of thousands of messages a second using ~250MiB of memory! DuckDB also supports a rich ecosystem of sinks and connectors! https://sql-flow.com/docs/category/tutorials/ https://github.com/turbolytics/sql-flow We were tired of running JVM's for simple stream processing, and also of bespoke one off stream processors I would love your feedback, criticisms and/or…
SQLFlow is a lightweight stream processing engine for Kafka that uses DuckDB to process tens of thousands of messages per second while consuming approximately 250MB of memory. It is designed for developers who need… Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes.
- 1
I'm a couple years late to the party but DuckDB is blowing my mind. I couldn't find an good embedded DuckDB stream processing solution so I hacked my own, calling it SQLFlow: https://github.com/turbolytics/sql-flow SQLFlow enables writing stream transformations in pure sql, executed using DuckDB. The goal was to create a lightweight, performant stream processing engine using pure SQL transformations, DuckDB didn't disappoint! I'd love your feedback, feature requests, impressions, or just comments. I'd love to turn this into a stable, usable project that people are…
2023 · github.com · its alternatives →
- 2
A flink alternative powered by DuckDB
2025 · turbolytics.io · its alternatives →
- 3
A Python stream processing engine modeled after Yahoo! Pipes - nerevu/riko
2016 · github.com · its alternatives →
- 4
Distributed Stream and Batch Processing. Contribute to hazelcast/hazelcast-jet development by creating an account on GitHub.
2020 · github.com · its alternatives →
- 5
tl;dr we built an embeddable stream processing engine in Rust using apache DataFusion, check us out at https://github.com/probably-nothing-labs/denormalized Hey HN, We’d like to showcase a very early version of our embeddable stream processing engine called Denormalized. The rise of DuckDB has abundantly made it clear that even for many workloads of Terabyte scale, a single node system outshines the distributed query engines of previous generation such as Spark, Snowflake etc in terms of both performance and cost. Now a lot of workloads DuckDB is used for were normally…
2024 · github.com · its alternatives →
- 6DU
I built Duck-UI, a web-based SQL editor that runs DuckDB entirely in your browser via WebAssembly. No backend required. The Problem: Every time I needed to query csv, parquet, or even to play with SQL, I had to either: (a) spin up a Jupyter notebook (b) use the CLI (c) upload to a hosted service. Friction at every step (TOO MUCH to load a csv or even to test some sql (study)... The Solution: DuckDB's WASM runtime lets us run SQL analysis client-side. Load CSV/JSON/Parquet files from disk or URL, write SQL, get results instantly. Data stays on your machine. What It Does: SQL editor…
2025 · demo.duckui.com · its alternatives →
- 7
DuckDB for streaming data. Contribute to turbolytics/sql-flow development by creating an account on GitHub.
2024 · github.com · its alternatives →
- 8
Hey HN, Arroyo is a modern, open-source stream processing engine, that lets anyone write complex queries on event streams just by writing SQL—windowing, aggregating, and joining events with sub-second latency. Today data processing typically happens in batch data warehouses like BigQuery and Snowflake despite the fact that most of the data is coming in as streams. Data teams have to build complex orchestration systems to handle late-arriving data and job failures while trying to minimize latency. Stream processing offers an alternative approach, where the query is compiled into a streaming…
2023 · github.com · its alternatives →
- 9
Hi HN! We are Ashish and Armend, founders of GlassFlow. We just launched our open-source streaming ETL that deduplicates and joins Kafka streams before ingesting them to ClickHouse https://github.com/glassflow/clickhouse-etl Why we built this: Dedup with batch data is straightforward. You load the data into a temporary table. Then, find only the latest versions of the record through hashes or keys and keep them. After that, move the clean data into your main table. But have you tried this with streaming data? Users of our prev product were running real-time analytics…
2025 · github.com · its alternatives →
- 10
TableFlow▲306Open-source CSV importer, alternative to Flatfile
2023 · tableflow.com · its alternatives →
- 11
Goka is a compact yet powerful distributed stream processing library for Apache Kafka written in Go. - lovoo/goka
2017 · github.com · its alternatives →
- 12
Kafka-based Job Queue for Python. Contribute to joowani/kq development by creating an account on GitHub.
2016 · github.com · its alternatives →
- 13
🦭 Distributed log streaming engine built from first principles - nubskr/walrus
Dec 2025 · github.com · its alternatives →
- 14RT
We are working on a product which makes stream processing easy for data analysts and data scientists. It’s a higher level alternative to Flink, Kafka Streams and Spark Streams. As well as making stream processing simpler, we also felt that lots of people were building these AWS stream processing platforms based on Kafka and something like Flink so wanted to offer this as a service. We would welcome any input on the space or approach: Home page: http://Timeflow.systems Demo: https://youtu.be/fGfgBaHXOkE
2020 · its alternatives →
- 15PS
Hello HN, I am Sai Srirampur, one of the Co-founders of PeerDB. (https://github.com/PeerDB-io/peerdb). We spent the past 7 months building a solid experience to replicate data from Postgres to data warehouses. Now we're expanding to queues. PeerDB Streams provides a simple and native way to replicate changes as they happen in Postgres to Queues (Kafka, Redpanda, Google PubSub, etc). We use Postgres logical decoding to enable Change Data Capture (CDC). Blog post here: https://blog.peerdb.io/peerdb-streams-simple-native-postgres.... 10-min quickstart here:…
2024 · its alternatives →
- 16LV
2016 · alooma.com · its alternatives →
- 17
- 18
Diskless Kafka® on S3. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency. Multi-AZ Availability. - AutoMQ/automq
2024 · github.com · its alternatives →
- 19
- 20
Hey HN, I'm really proud to share with you my new open source project: Kestra https://github.com/kestra-io/kestra I created a few years ago a successful open source AKHQ project: https://github.com/tchiotludo/akhq (renamed from KafkaHQ) which has been adopted by big companies like Best Buy, Pipedrive, BMW, Decathlon and many more. 2300 stars, 120 contributors, 10M docker downloads, much more than I expected. Now let's talk about Kestra, an infinitely scalable orchestration and scheduling platform for creating, running, scheduling and monitoring…
2022 · github.com · its alternatives →
- 21
Open-Source Web UI for Apache Kafka Management. Contribute to provectus/kafka-ui development by creating an account on GitHub.
2021 · github.com · its alternatives →
- 22
Hi HN - I’m Peter, here with Harry (devhawk), and we’re building DBOS Java, an open-source Java library for durable workflows, backed by Postgres. https://github.com/dbos-inc/dbos-transact-java Essentially, DBOS helps you write long-lived, reliable code that can survive failures, restarts, and crashes without losing state or duplicating work. As your workflows run, it checkpoints each step they take in a Postgres database. When a process stops (fails, restarts, or crashes), your program can recover from those checkpoints to restore its exact state and continue from where…
Nov 2025 · github.com · its alternatives →
- 23
I started thinking about this approach after working on a large-scale project for a major financial company where our group developed a distributed in-house data processing solution. On a regular basis, it ingested a few gigabytes of financial data and, within a tight SLA time limit, produced a lot of enriched/aggregated/validated data for a number of customers. Sometimes, source data had errors, so operators with domain knowledge had to verify data validity at some checkpoints, immediately make corrections, and re-run parts of the workflow manually. The solution involved complex…
2023 · capillaries.io · its alternatives →
- 24
A highly configurable Google Cloud Dataflow pipeline that writes data into Google Big Query table from Pub/Sub - bomboradata/pubsub-to-bigquery
2017 · github.com · its alternatives →
Also compare
- SQLFlow: SQL Stream Processing Using DuckDB alternatives
- SQLFlow - Easy Stream Processing alternatives
- Riko – A Python stream processing engine modeled after Yahoo! Pipes alternatives
- Jet – in-memory, fault-tolerant, distributed stream processing alternatives
- Denormalized – Embeddable Stream Processing in Rust and DataFusion alternatives
- Duck-UI – Browser-Based SQL IDE for DuckDB alternatives
Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →