Alternatives
Products that do what 7x faster Iceberg ingestion, how we redesigned OLake's writer does
OLake is our open-source tool for ingesting Database & Kafka data into Apache Iceberg. We recently redesigned the write pipeline and saw ~7x throughput improvements. Sharing the architecture decisions, trade-offs, and benchmarks.
- 1OO
Hi HN, Today we’re excited to introduce OLake (github.com/datazip-inc/olake, 130+ and growing fast), an open-source tool built to help you replicate Database (MongoDB, for now, mysql and postgres under development) data into Data Lakehouse at faster speed without any hassle of managing Debezium or kafka (at least 10x faster than Airbyte and Fivetran at fraction of the cost, refer docs for benchmarks - https://olake.io/docs/connectors/mongodb/benchmarks). You might think “we don't need yet another ETL tool”, true but we tried existing tools (proprietary…
2025
- 2IB
Hi there, Burak here. I built an open-source data copy tool called ingestr (https://github.com/bruin-data/ingestr) I did build quite a few data warehouses both for the companies I worked at, as well as for consultancy projects. One of the more common pain points I observed was that everyone had to rebuild the same data ingestion bit over and over again, and each in different ways: - some wrote code for the ingestion from scratch to various degrees - some used off-the-shelf data ingestion tools like Fivetran / Airbyte I have always disliked both of these approaches,…
2024 · github.com
- 3SO
We just released a major upgrade to SigNoz with support for Logs management based on ClickHouse. Would love to get any feedback from the community here on what you think and any questions you may have for us Many big companies like Uber and Cloudflare have been shifting to ClickHouse as their main workhorse for Logs management seeing much better performance. for e.g Cloudflare recently shifted from Elastic to ClickHouse and are seeing 8x improvement in memory/cpu resource requirement in ingestion. This is our first release with Logs support and we have added support for: - Filtering…
2022 · github.com
- 4OS
2022 · github.com
- 5IB
Every data pipeline job I had to tackle required quite a few components to set up: - One tool to ingest data - Another one to transform it - If you wanted to run Python, set up an orchestrator - If you need to check the data, a data quality tool Let alone this being hard to set up and taking time, it is also pretty high-maintenance. I had to do a lot of infra work, and while this being billable hours for me I didn’t enjoy the work at all. For some parts of it, there were nice solutions like dbt, but in the end for an end-to-end workflow, it didn’t work. That’s why I decided to build an…
2024 · github.com
- 6
- 7SO
Hello! For the past year I’ve been working on a fully-managed data warehouse built on Clickhouse. I built this because I was frustrated with how much work was required to run an OLAP database in prod: re-writing my app to do batch inserts, managing clusters and needing to look up special CREATE TABLE syntax every time I made a change. I found pricing for other warehouses confusing (what is a “credit” exactly?) and worried about getting capacity-planning wrong. I was previously building accounting software for firms with millions of transactions. I desperately needed to move from Postgres to…
2023 · github.com
- 8
- 9KA
2020 · github.com
- 10SA
2022 · shalerb.org
- 11OA
I’ve been a long-time Snowplow user and unofficial evangelizer. I have deep respect for its founders, Alex and Yali, who I met a few times. What made me fall in love with Snowplow was that it was unopinionated, gave access to raw event data, and was truly open source. Back in 2013, that changed everything for me. I couldn’t look at GA the same way again. Over the years, analytics moved into SQL warehouses driven by cheaper CPU/storage, dbt, reproducibility, and transparency. I saw the need for a democratized Snowplow pipeline and launched a hosted version in 2019. In January 2024,…
Oct 2025 · opensnowcat.io
- 12PS
Hey HN! Lev here, author of PgDog (https://github.com/pgdogdev/pgdog). I’m scaling our favorite database, PostgreSQL. PgDog is a new open source proxy, written in Rust, with first-class support for sharding — without changes to your app or needing database extensions. Here’s a walkthrough of how it works: https://www.youtube.com/watch?v=y6sebczWZ-c Running Postgres at scale is hard. Eventually, one primary isn’t enough at which point you need to split it up. Since there is currently no good tooling out there to do this, teams end up breaking their apps…
2025 · github.com
- 13
- 14OS
Hey HN! We’ve built Pretzel, an open-source data exploration and visualization tool that runs fully in the browser and can handle large files (200 MB CSV on my 8gb MacBook air is snappy). It’s also reactive - so if, for example, you change a filter, all the data transform blocks after it re-evaluate automatically. You can try it here: https://pretzelai.github.io/ (static hosted webpage) or see a demo video here: https://www.youtube.com/watch?v=73wNEun_L7w You can play with the demo CSV that’s pre-loaded (GitHub data of text-editor adjacent projects) or upload…
2024 · github.com
- 15FD
2023 · flakehub.com
- 16DF
Hello Everyone! We built SQLFlow as a lightweight stream processing engine. We leverage DuckDB as the stream processing engine, which gives SQLFlow the ability to process 10's of thousands of messages a second using ~250MiB of memory! DuckDB also supports a rich ecosystem of sinks and connectors! https://sql-flow.com/docs/category/tutorials/ https://github.com/turbolytics/sql-flow We were tired of running JVM's for simple stream processing, and also of bespoke one off stream processors I would love your feedback, criticisms and/or…
Dec 2025 · sql-flow.com
- 17SS
May 2026 · github.com
- 18SL
At Arcesium, we needed a way to run SQL-driven CRUD operations, manage slowly-changing dimensions, and handle schema evolution on our cloud data lakes—without the overhead of spinning up distributed clusters or relying on vendor-managed query services. That’s why we built SwiftLake, a lightweight open-source library that brings Apache Iceberg and DuckDB together: - Lightweight & cluster-free: Runs as a single Java service with embedded DuckDB—no distributed infra required. - Full CRUD and SCD1/SCD2 support: Reads, inserts/appends, updates, deletes, and merges with ACID guarantees.…
Sep 2025 · github.com
- 19IM
Hey HN, I'm Adithya, a 20-year-old dev from India. I have been working with GenAI for the past year, and I've found it really painful to deal with the many different forms of data out there and get the best representation of it for my AI applications. That's why I built OmniParse—an open-source platform designed to handle any unstructured data and transform it into optimized, structured representations. Key Features: - Completely local processing—no external APIs - Supports ~20 file types - Converts documents, multimedia, and web pages to high-quality structured markdown - Table extraction,…
2024 · github.com
- 20GD
2017 · github.com
- 21OE
Hello folks, We are launching OpenObserve. An open source Elasticsearch/Splunk/Datadog alternative written in rust and vue that is super easy to get started with and has 140x lower storage cost compared to elasticsearch. It offers logs, metrics, traces, dashboards, alerts, functions (run aws lambda like functions during ingestion and query to enrich, redact, transform, normalize and whatever else you want to do. Think redacting email IDs from logs, adding geolocation based on IP address, etc). You can do all of this from the UI, no messing up with configuration files. OpenObserve…
2023 · github.com
- 22PB
Hi HN! This is an early proof of concept of a branchable, versioned OLTP + OLAP database that runs on a single, open copy of your data in object storage. If you are familiar with Databricks' LTAP (https://www.databricks.com/company/newsroom/press-releases/d...) announcement from June, you can think of this as aspiring to be a fully open source, Apache 2.0 LTAP alternative with additional data versioning/auditability guarantees that enable audit, as_of queries, and (eventually) revert straight out of the box. How it works: 1. Writes land in vanilla postgres…
Jul 2026 · github.com
- 23UF
2021 · github.com
- 24GO
Hi HN! We are Ashish and Armend, founders of GlassFlow. We just launched our open-source streaming ETL that deduplicates and joins Kafka streams before ingesting them to ClickHouse https://github.com/glassflow/clickhouse-etl Why we built this: Dedup with batch data is straightforward. You load the data into a temporary table. Then, find only the latest versions of the record through hashes or keys and keep them. After that, move the clean data into your main table. But have you tried this with streaming data? Users of our prev product were running real-time analytics…
2025 · github.com
Ranked by how close each launch is in meaning, then by votes. Refine with a description →