7x faster Iceberg ingestion, how we redesigned OLake's writer
OLake is our open-source tool for ingesting Database & Kafka data into Apache Iceberg. We recently redesigned the write pipeline and saw ~7x throughput improvements. Sharing the architecture decisions, trade-offs, and benchmarks.
7x faster Iceberg ingestion, how we redesigned OLake's writer launched on December 5, 2025 with 5 votes, #2,550 of 11,628 launches that month and more than 75% of that year's launches.
What it does
Data ingestion performance is critical when you are writing in Iceberg format to data lake. When your pipeline becomes a bottleneck, it affects everything downstream—from real-time analytics to machine learning workflows. We started with OneStack in Datazip to solve the problem of Data Analytics, but then we were bound to see the problem on Data Ingestion itself. To solve this problem for iceberg native writes we built OLake Go. At first we created basic Iceberg writer, but then we saw the issues with it (I will be writing about the issues in a new section). Today we have resolved all those bottlenecks. The result? A 7× performance improvement in Apache Iceberg, without the complexity of…from olake.io
Does a similar job
all alternatives →- OOOLake[open source] Fastest database to Iceberg data replication tool2025 · ▲14
Hi HN, Today we’re excited to introduce OLake (github.com/datazip-inc/olake, 130+ and growing fast), an open-source tool built to help you replicate Database (MongoDB, for now, mysql and postgres under development) data into Data Lakehouse at faster speed without any hassle of managing Debezium or kafka (at least 10x faster than Airbyte and Fivetran at fraction of the cost, refer docs for benchmarks - https://olake.io/docs/connectors/mongodb/benchmarks). You might think “we don't need yet another ETL tool”, true but we tried existing tools (proprietary…
I built an open-source data copy tool called ingestr2024 · github.com · ▲156Hi there, Burak here. I built an open-source data copy tool called ingestr (https://github.com/bruin-data/ingestr) I did build quite a few data warehouses both for the companies I worked at, as well as for consultancy projects. One of the more common pain points I observed was that everyone had to rebuild the same data ingestion bit over and over again, and each in different ways: - some wrote code for the ingestion from scratch to various degrees - some used off-the-shelf data ingestion tools like Fivetran / Airbyte I have always disliked both of these approaches,…
SigNoz – open-source alternative to DataDog, NewRelic2022 · github.com · ▲510We just released a major upgrade to SigNoz with support for Logs management based on ClickHouse. Would love to get any feedback from the community here on what you think and any questions you may have for us Many big companies like Uber and Cloudflare have been shifting to ClickHouse as their main workhorse for Logs management seeing much better performance. for e.g Cloudflare recently shifted from Elastic to ClickHouse and are seeing 8x improvement in memory/cpu resource requirement in ingestion. This is our first release with Logs support and we have added support for: - Filtering…
I built an open-source data pipeline tool in Go2024 · github.com · ▲200Every data pipeline job I had to tackle required quite a few components to set up: - One tool to ingest data - Another one to transform it - If you wanted to run Python, set up an orchestrator - If you need to check the data, a data quality tool Let alone this being hard to set up and taking time, it is also pretty high-maintenance. I had to do a lot of infra work, and while this being billable hours for me I didn’t enjoy the work at all. For some parts of it, there were nice solutions like dbt, but in the end for an end-to-end workflow, it didn’t work. That’s why I decided to build an…
Open-source serverless security lake powered by Rust + Apache Iceberg2022 · github.com · ▲63Open source security data lake for threat hunting, detection & response, and cybersecurity analytics at petabyte scale on AWS - matanolabs/matano
More dev tools this month
the category →
Contribute to oosexgang/hackios-xnu-kernel development by creating an account on GitHub.
Dev tools · 2d ago · github.com
open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.
Dev tools · 30d ago · github.com
Contribute to reladraw/reladraw development by creating an account on GitHub.
Dev tools · 28d ago · github.com
OpenBot▲362Grok Bot alternative: free, local, open-source, multiplayer
Dev tools · 25d ago · openbot.run
Talanoa▲346Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…
Dev tools · 22d ago · talanoa.email
Source: Show HN post ↗
Launched alongside, December 2025
the whole month →- GP
Life & fun · Dec 2025 · dosaygo-studio.github.io
Hi everyone! My name's Luke and I made the original Jmail here alongside Riley Walz. We had a ton of friends collaborate on building out more of the app suite last night in lieue of DOJ's "Epstein files" release. Please AMA!
Life & fun · Dec 2025 · jmail.world
PlanEat AI▲733AI turns your health goals into a 7-day menu & grocery list
AI · Dec 2025 · planeatai.com
Community, All the HN belong to you. This is an archive of hacker news that fits in your browser. When I made HN Made of Primes I realized I could probably do this offline sqlite/wasm thing with the whole GBs of archive. The whole dataset. So I tried it, and this is it. Have Hacker News on your device. Go to this repo (https://github.com/DOSAYGO-STUDIO/HackerBook): you can download it. Big Query -> ETL -> npx serve docs - that's it. 20 years of HN arguments and beauty, can be yours forever. So they'll never die. Ever. It's the unkillable static archive of HN and it's…
Dev tools · Dec 2025 · hackerbook.dosaygo.com
