GlassFlow – OSS streaming dedup and joins from Kafka to ClickHouse

In plain words

GlassFlow is an open-source streaming ETL tool that deduplicates and joins Kafka streams before ingesting them into ClickHouse. It addresses the challenge of handling duplicate records in real-time data pipelines, which commonly occur when data flows from multiple sources like CRMs and clickstream systems. The tool is designed for data engineers and analytics teams building real-time analytics pipelines who need to ensure data quality and accuracy before loading into ClickHouse.

written from the facts on this page · September 2026

From the sources

GlassFlow ETL: Open source data ingestion and transformations for ClickHouse pipelines - glassflow/clickhouse-etl

GlassFlow ClickHouse ETL is an open-source project by GlassFlow . Its documentation lives at glassflow-etl.com . Looking for GlassFlow's current product? Visit glassflow.ai . Backfill historical data, keep CDC in sync, handle schema changes, normalize messy data, and keep ClickHouse queries correct. GlassFlow is an open-source stream processing engine designed for data ingestion and transformation from multiple sources into ClickHouse. GlassFlow comes with the following core functionalities: GlassFlow is open source and can be self-hosted on Kubernetes. GlassFlow works with any managed Kubernetes service like AWS EKS, GKE, AKS, and more. For detailed documentation, visit glassflow-etl.com .…from github.com

In the maker’s words, at launch

Hi HN! We are Ashish and Armend, founders of GlassFlow. We just launched our open-source streaming ETL that deduplicates and joins Kafka streams before ingesting them to ClickHouse https://github.com/glassflow/clickhouse-etl Why we built this: Dedup with batch data is straightforward. You load the…

• 
  SigNoz – open-source alternative to DataDog, NewRelic2022 · github.com · ▲510

  We just released a major upgrade to SigNoz with support for Logs management based on ClickHouse. Would love to get any feedback from the community here on what you think and any questions you may have for us Many big companies like Uber and Cloudflare have been shifting to ClickHouse as their main workhorse for Logs management seeing much better performance. for e.g Cloudflare recently shifted from Elastic to ClickHouse and are seeing 8x improvement in memory/cpu resource requirement in ingestion. This is our first release with Logs support and we have added support for: - Filtering…

• 
  ClickHouse-local – a small tool for serverless data analytics2023 · clickhouse.com · ▲289

  Me with my friend Vitaly Ludvichenko made an experiment to combine ClickHouse server and client to make a self-contained program running a database engine and processing data without a server: https://github.com/ClickHouse/ClickHouse/pull/150 Development continued in the past 6 years, and now clickhouse-local becomes a swiss-army knife for data processing. Say "ffmpeg" for datasets and more. It can resemble textql, octosql, dsq, duckdb, trdsql, q, datafusion-cli, spyql, but has better capabilities and performance. Here is a tutorial:…

• 
  ScratchDB – Open-Source Snowflake on ClickHouse2023 · github.com · ▲261

  Hello! For the past year I’ve been working on a fully-managed data warehouse built on Clickhouse. I built this because I was frustrated with how much work was required to run an OLAP database in prod: re-writing my app to do batch inserts, managing clusters and needing to look up special CREATE TABLE syntax every time I made a change. I found pricing for other warehouses confusing (what is a “credit” exactly?) and worried about getting capacity-planning wrong. I was previously building accounting software for firms with millions of transactions. I desperately needed to move from Postgres to…

• 
  DuckDB for Kafka Stream ProcessingDec 2025 · sql-flow.com · ▲77

  Hello Everyone! We built SQLFlow as a lightweight stream processing engine. We leverage DuckDB as the stream processing engine, which gives SQLFlow the ability to process 10's of thousands of messages a second using ~250MiB of memory! DuckDB also supports a rich ecosystem of sinks and connectors! https://sql-flow.com/docs/category/tutorials/ https://github.com/turbolytics/sql-flow We were tired of running JVM's for simple stream processing, and also of bespoke one off stream processors I would love your feedback, criticisms and/or…

• 
  DoubleCloud2022 · double.cloud · ▲168

  Get the most out of your data in the cloud

• 
  TableFlow2023 · tableflow.com · ▲306

  Open-source CSV importer, alternative to Flatfile

• 
  Run full iOS 16.5 firmware on Android Devices▲515

  Contribute to oosexgang/hackios-xnu-kernel development by creating an account on GitHub.

  Dev tools · 2d ago · github.com

• 
  /dev/null▲492

  a social network for devs

  Dev tools · 5d ago · dev-null.sh

• 
  Whiteboard (YC W26) – an open-source IDE for thoughtful software design▲423

  open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.

  Dev tools · 29d ago · github.com

• 
  Reladraw – A diagram language where you decide where to place things▲411

  Contribute to reladraw/reladraw development by creating an account on GitHub.

  Dev tools · 27d ago · github.com

• 
  OpenBot▲362

  Grok Bot alternative: free, local, open-source, multiplayer

  Dev tools · 25d ago · openbot.run

• 
  Talanoa▲346

  Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…

  Dev tools · 22d ago · talanoa.email

• 
  Tapflow 2.0▲1,125

  We built Tapflow for devs, designers, PMs, and tech folks who’ve been sitting on internal docs, templates, and systems — but never productized them. Upload a PDF, notes, or workshop deck, and Tapflow drafts a structured product (with chapters, teaser and pricing plans). It’s good for AI kits, onboarding guides, design systems, and frameworks. You can sell through Stripe, LemonSqueezy, or PayPal. Real-time collaboration is in early beta. No site builder, no LMS — clean design and just focused tools to ship. Fully bootstrapped and just launched on PH today →…

  Work · 2025 · tapflow.ai

• C9
  Clippy – 90s UI for local LLMs▲1,122

  Life & fun · 2025 · felixrieseberg.github.io

• 
  Appwrite Sites▲990

  Appwrite 2.0 brings a new engine, a rebuilt Console, five database types, an S3 API, an OAuth 2.1 server, and organization-level Domains and Firewall.

  Dev tools · 2025 · appwrite.io

• 
  AI Meeting Notes by Notion▲965

  Perfect meeting memory in Notion

  AI · 2025 · notion.com

• 
  Den▲948

  Multiplayer AI agent builder

  AI · 2025 · getden.io

• 
  Raycast for iOS▲870

  Powerful productivity on the go

  AI · 2025 · raycast.com