Alternatives
Products that do what AutoMQ 1.5.0 released, providing native Iceberg support for Kafka does
AutoMQ is a fully open-source, next-generation Kafka built on top of S3. Version 1.5.0 is a very important release. In March of this year, Confluent officially launched the commercial capability of Tableflow. Now, with the AutoMQ open-source software, you can also experience this killer feature. Simply put, for Kafka streaming data, AutoMQ can automatically store it in S3 in Iceberg Table format, so you no longer need to manage Flink Jobs and Spark Jobs yourself to perform ETL operations and convert Kafka data into table format. We believe this will be the new paradigm for Kafka stream data…
- 1DF
Hello Everyone! We built SQLFlow as a lightweight stream processing engine. We leverage DuckDB as the stream processing engine, which gives SQLFlow the ability to process 10's of thousands of messages a second using ~250MiB of memory! DuckDB also supports a rich ecosystem of sinks and connectors! https://sql-flow.com/docs/category/tutorials/ https://github.com/turbolytics/sql-flow We were tired of running JVM's for simple stream processing, and also of bespoke one off stream processors I would love your feedback, criticisms and/or…
Dec 2025 · sql-flow.com
- 2
- 3GO
Hi HN! We are Ashish and Armend, founders of GlassFlow. We just launched our open-source streaming ETL that deduplicates and joins Kafka streams before ingesting them to ClickHouse https://github.com/glassflow/clickhouse-etl Why we built this: Dedup with batch data is straightforward. You load the data into a temporary table. Then, find only the latest versions of the record through hashes or keys and keep them. After that, move the clean data into your main table. But have you tried this with streaming data? Users of our prev product were running real-time analytics…
2025 · github.com
- 4
- 5
- 6SS
May 2026 · github.com
- 7SS
Hey HN, I built StreamHouse, an open-source streaming platform that replaces Kafka's broker-managed storage with direct S3 writes. The goal: same semantics, fraction of the cost. How it works: Producers batch and compress records, a stateless server manages partition routing and metadata (SQLite for dev, PostgreSQL for prod), and segments land directly in S3. Consumers read from S3 with a local segment cache. No broker disks to manage, no replication factor to tune — S3 gives you 11 nines of durability out of the box. What's there today: - Producer API with batching, LZ4 compression, and…
Feb 2026 · github.com
- 8
- 9CK
FastStream - https://github.com/airtai/faststream, a stream processing framework, already supports Kafka stream processing using the aiokafka library, as well as other brokers such as Redis, RabbitMQ, and NATS. Responding to popular demand, the latest 0.4.0rc0 version introduces support for Kafka stream processing using Confluent Kafka's Python library - https://github.com/confluentinc/confluent-kafka-python. Please take a look at it and let us know what you think: https://faststream.airt.ai/0.4/confluent/ Wondering why and how…
2024
- 10
- 11KS
2016 · github.com
- 12
- 13PD
PicoMQ is a Rust server for Durable Streams, built on Object Store. Cheap, URL-addressable, granular streams (create/append/read/long-poll/SSE), with Pico Protocol or Durable Streams Protocol as the facade. S3Stream is the stream storage primitive, used in AutoMQ, shipped as a Rust library. Coordination is a command log in Postgres.
13d ago · picomq.com
- 147F
OLake is our open-source tool for ingesting Database & Kafka data into Apache Iceberg. We recently redesigned the write pipeline and saw ~7x throughput improvements. Sharing the architecture decisions, trade-offs, and benchmarks.
Dec 2025 · olake.io
- 15MK
I built a Model Context Protocol server that allows users to interact with Kafka using natural language instead of complex CLIs. It uses LLMs to interpret commands in plain English and executes the corresponding Kafka operations. This makes Kafka more accessible to non-experts while maintaining its powerful functionality.
2025 · github.com
- 16B5
I built a message broker that speaks the Kafka wire protocol, so any Kafka client (librdkafka, kafka-python, kcat, etc.) works without code changes. The entire binary is 52KB. No JVM, no ZooKeeper, no third-party libraries — just C++20 with kqueue/epoll. Starts in <10ms, uses 0% CPU when idle. I built this because running Kafka locally for development is painful — gigabytes of RAM, slow startup, ZooKeeper/KRaft configuration. I just wanted something that accepts produce requests and gets out of the way. Technical details: - Single-threaded event loop (kqueue on macOS, epoll on…
Feb 2026 · github.com
- 17SL
At Arcesium, we needed a way to run SQL-driven CRUD operations, manage slowly-changing dimensions, and handle schema evolution on our cloud data lakes—without the overhead of spinning up distributed clusters or relying on vendor-managed query services. That’s why we built SwiftLake, a lightweight open-source library that brings Apache Iceberg and DuckDB together: - Lightweight & cluster-free: Runs as a single Java service with embedded DuckDB—no distributed infra required. - Full CRUD and SCD1/SCD2 support: Reads, inserts/appends, updates, deletes, and merges with ACID guarantees.…
Sep 2025 · github.com
- 18LA
Hi HN. I'm Carl, creator of LavinMQ. LavinMQ is an open-source message broker. AMQP 0-9-1, MQTT, HTTP and streaming. Single binary, minimal resource use. If you know RabbitMQ, it's that (but running on a fraction of hardware). We're the team behind CloudAMQP where we've hosted RabbitMQ for 14 years. Sometimes customers hit issues we couldn't fully explain / work around. So we built our own broker, in order to provide solutions where we previously couldn't (which is much easier when you control the full stack). It all started by me building an open-source AMQP proxy to handle short-lived…
Apr 2026 · github.com
- 19OS
Existing OSS C++ projects like ClickHouse and DuckDB support reading from Iceberg tables. Writing requires Spark, PyIceberg, or managed services. In this PR https://github.com/timeplus-io/proton/pull/928, we are open-sourcing a C++ implementation of Iceberg integration. It's an MVP, focusing on REST catalog and S3 read/write(S3 table support coming soon). You can use Timeplus to continuously read data from MSK and stream writes to S3 in the Iceberg format. No JVM. No Python. Just a low-overhead, high-throughput C++ engine. Docker/K8s are optional. Demo…
2025 · github.com
- 20KS
2020 · github.com
- 21
- 22IO
Today we're (@sprosvirnin and I) launching the first open-source release of IceGate - a Rust-native engine that stores logs (metrics and traces coming soon) in Apache Iceberg tables on your own object storage. Observability costs are out of control. Teams pay per GB ingested, get locked into proprietary formats, and can't use their own data outside the vendor's UI. We built the first observability data lake engine to fix this. This is our first release under Apache 2.0. We're building in the open and looking for contributors and early adopters who want observability without the markup. Star…
Apr 2026 · github.com
- 23CM
Hi HN! We are excited to share something we have been building for the last 12 months. Castled is a warehouse-native marketing platform built directly on cloud data warehouses like Snowflake, BigQuery, Redshift, and Postgres. Castled allows you to directly use the customer data in your data warehouse and engage them across different channels like Email, Sms, WhatsApp, push, and In-app - without having to copy the data to another tool. We started our journey by building an open-source Reverse ETL solution to make the warehouse data actionable to marketers. However, after talking to 100s of…
2023 · castled.io
- 24DS
Hey, I'm a co-founder at ElectricSQL. Durable Streams is the delivery protocol underneath our Postgres sync engine—we've been refining it in production for 18 months. The core idea: streams get their own URL and use opaque, monotonic offsets. Clients persist the last offset they processed and resume with "give me everything after X." No server-side session state, CDN-cacheable, plain HTTP. We kept seeing teams reinvent this for AI token streaming and real-time apps, so we're standardizing it as a standalone protocol. The repo has a reference Node.js server and TypeScript client. Would love…
Dec 2025 · github.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →