Dev tools · alternatives · 2026
24 alternatives to 7x faster Iceberg ingestion, how we redesigned OLake's writer
OLake is our open-source tool for ingesting Database & Kafka data into Apache Iceberg. We recently redesigned the write pipeline and saw ~7x throughput improvements. Sharing the architecture decisions, trade-offs, and benchmarks.
Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes.
- 1OO
Hi HN, Today we’re excited to introduce OLake (github.com/datazip-inc/olake, 130+ and growing fast), an open-source tool built to help you replicate Database (MongoDB, for now, mysql and postgres under development) data into Data Lakehouse at faster speed without any hassle of managing Debezium or kafka (at least 10x faster than Airbyte and Fivetran at fraction of the cost, refer docs for benchmarks - https://olake.io/docs/connectors/mongodb/benchmarks). You might think “we don't need yet another ETL tool”, true but we tried existing tools (proprietary…
2025 · its alternatives →
- 2
Hi there, Burak here. I built an open-source data copy tool called ingestr (https://github.com/bruin-data/ingestr) I did build quite a few data warehouses both for the companies I worked at, as well as for consultancy projects. One of the more common pain points I observed was that everyone had to rebuild the same data ingestion bit over and over again, and each in different ways: - some wrote code for the ingestion from scratch to various degrees - some used off-the-shelf data ingestion tools like Fivetran / Airbyte I have always disliked both of these approaches,…
2024 · github.com · its alternatives →
- 3
We just released a major upgrade to SigNoz with support for Logs management based on ClickHouse. Would love to get any feedback from the community here on what you think and any questions you may have for us Many big companies like Uber and Cloudflare have been shifting to ClickHouse as their main workhorse for Logs management seeing much better performance. for e.g Cloudflare recently shifted from Elastic to ClickHouse and are seeing 8x improvement in memory/cpu resource requirement in ingestion. This is our first release with Logs support and we have added support for: - Filtering…
2022 · github.com · its alternatives →
- 4
Every data pipeline job I had to tackle required quite a few components to set up: - One tool to ingest data - Another one to transform it - If you wanted to run Python, set up an orchestrator - If you need to check the data, a data quality tool Let alone this being hard to set up and taking time, it is also pretty high-maintenance. I had to do a lot of infra work, and while this being billable hours for me I didn’t enjoy the work at all. For some parts of it, there were nice solutions like dbt, but in the end for an end-to-end workflow, it didn’t work. That’s why I decided to build an…
2024 · github.com · its alternatives →
- 5
Open source security data lake for threat hunting, detection & response, and cybersecurity analytics at petabyte scale on AWS - matanolabs/matano
2022 · github.com · its alternatives →
- 6
Hello! For the past year I’ve been working on a fully-managed data warehouse built on Clickhouse. I built this because I was frustrated with how much work was required to run an OLAP database in prod: re-writing my app to do batch inserts, managing clusters and needing to look up special CREATE TABLE syntax every time I made a change. I found pricing for other warehouses confusing (what is a “credit” exactly?) and worried about getting capacity-planning wrong. I was previously building accounting software for firms with millions of transactions. I desperately needed to move from Postgres to…
2023 · github.com · its alternatives →
- 7
- 8
- 9SA
2022 · shalerb.org · its alternatives →
- 10
Open-Source Web UI for Apache Kafka Management. Contribute to provectus/kafka-ui development by creating an account on GitHub.
2021 · github.com · its alternatives →
- 11
I’ve been a long-time Snowplow user and unofficial evangelizer. I have deep respect for its founders, Alex and Yali, who I met a few times. What made me fall in love with Snowplow was that it was unopinionated, gave access to raw event data, and was truly open source. Back in 2013, that changed everything for me. I couldn’t look at GA the same way again. Over the years, analytics moved into SQL warehouses driven by cheaper CPU/storage, dbt, reproducibility, and transparency. I saw the need for a democratized Snowplow pipeline and launched a hosted version in 2019. In January 2024,…
2025 · opensnowcat.io · its alternatives →
- 12
Open-source, versatile, fast and lightweight UI for Kafka
2023 · github.com · its alternatives →
- 13
2023 · flakehub.com · its alternatives →
- 14
Hello Everyone! We built SQLFlow as a lightweight stream processing engine. We leverage DuckDB as the stream processing engine, which gives SQLFlow the ability to process 10's of thousands of messages a second using ~250MiB of memory! DuckDB also supports a rich ecosystem of sinks and connectors! https://sql-flow.com/docs/category/tutorials/ https://github.com/turbolytics/sql-flow We were tired of running JVM's for simple stream processing, and also of bespoke one off stream processors I would love your feedback, criticisms and/or…
Dec 2025 · sql-flow.com · its alternatives →
- 15
go-stash is a high performance, free and open source server-side data processing pipeline that ingests data from Kafka, processes it, and then sends it to ElasticSearch. - kevwan/go-stash
2021 · github.com · its alternatives →
- 16
At Arcesium, we needed a way to run SQL-driven CRUD operations, manage slowly-changing dimensions, and handle schema evolution on our cloud data lakes—without the overhead of spinning up distributed clusters or relying on vendor-managed query services. That’s why we built SwiftLake, a lightweight open-source library that brings Apache Iceberg and DuckDB together: - Lightweight & cluster-free: Runs as a single Java service with embedded DuckDB—no distributed infra required. - Full CRUD and SCD1/SCD2 support: Reads, inserts/appends, updates, deletes, and merges with ACID guarantees.…
2025 · github.com · its alternatives →
- 17
2015 · github.com · its alternatives →
- 18
Hey HN, I'm Adithya, a 20-year-old dev from India. I have been working with GenAI for the past year, and I've found it really painful to deal with the many different forms of data out there and get the best representation of it for my AI applications. That's why I built OmniParse—an open-source platform designed to handle any unstructured data and transform it into optimized, structured representations. Key Features: - Completely local processing—no external APIs - Supports ~20 file types - Converts documents, multimedia, and web pages to high-quality structured markdown - Table extraction,…
2024 · github.com · its alternatives →
- 19
Redpanda Console is a developer-friendly UI for managing your Kafka/Redpanda workloads. Console gives you a simple, interactive approach for gaining visibility into your topics, masking data, managing consumer groups, and exploring real-time data with time-travel debugging. - redpanda-data/console
2020 · github.com · its alternatives →
- 20
🦭 Distributed log streaming engine built from first principles - nubskr/walrus
Dec 2025 · github.com · its alternatives →
- 21
Goka is a compact yet powerful distributed stream processing library for Apache Kafka written in Go. - lovoo/goka
2017 · github.com · its alternatives →
- 22
Hi HN! This is an early proof of concept of a branchable, versioned OLTP + OLAP database that runs on a single, open copy of your data in object storage. If you are familiar with Databricks' LTAP (https://www.databricks.com/company/newsroom/press-releases/d...) announcement from June, you can think of this as aspiring to be a fully open source, Apache 2.0 LTAP alternative with additional data versioning/auditability guarantees that enable audit, as_of queries, and (eventually) revert straight out of the box. How it works: 1. Writes land in vanilla postgres…
Jul 2026 · github.com · its alternatives →
- 23LV
2016 · alooma.com · its alternatives →
- 24
Hi HN! We are Ashish and Armend, founders of GlassFlow. We just launched our open-source streaming ETL that deduplicates and joins Kafka streams before ingesting them to ClickHouse https://github.com/glassflow/clickhouse-etl Why we built this: Dedup with batch data is straightforward. You load the data into a temporary table. Then, find only the latest versions of the record through hashes or keys and keep them. After that, move the clean data into your main table. But have you tried this with streaming data? Users of our prev product were running real-time analytics…
2025 · github.com · its alternatives →
Also compare
- OLake[open source] Fastest database to Iceberg data replication tool alternatives
- I built an open-source data copy tool called ingestr alternatives
- SigNoz – open-source alternative to DataDog, NewRelic alternatives
- I built an open-source data pipeline tool in Go alternatives
- Open-source serverless security lake powered by Rust + Apache Iceberg alternatives
- ScratchDB – Open-Source Snowflake on ClickHouse alternatives
Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →