500k+ events/sec transformations for ClickHouse ingestion
Hi HN! We are Ashish and Armend, founders of GlassFlow. Over the last year, we worked with teams running high-throughput pipelines into self-hosted ClickHouse. Mostly for observability and real-time analytics. A question that came repeatedly was: What happens when throughput grows? Usually, things work fine at 10k events/sec, but we started seeing backpressure and errors at >100k. When the throughput per pipeline stops scaling, then adding more CPU/memory doesn’t help because often parts of the pipeline are not parallelized or are bottlenecked by state handling. At this point,…
In plain words
GlassFlow is a tool for building high-throughput data transformation pipelines into self-hosted ClickHouse, handling over 500,000 events per second. It addresses the challenge of scaling beyond 100k events per second, where traditional approaches like adding CPU or running multiple pipeline instances create bottlenecks and duplication. The platform is designed for teams running observability and real-time analytics workloads who need to process large data volumes without the complexity of managing parallel pipelines or stateful logic across multiple instances.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
Hi HN! We are Ashish and Armend, founders of GlassFlow. Over the last year, we worked with teams running high-throughput pipelines into self-hosted ClickHouse. Mostly for observability and real-time analytics. A question that came repeatedly was: What happens when throughput grows? Usually, things work fine at 10k events/sec, but we started seeing backpressure and errors at >100k. When the throughput per pipeline stops scaling, then adding more CPU/memory doesn’t help because often parts of the pipeline are not parallelized or are bottlenecked by state handling. At this point, engineers usually scale by adding more pipeline instances. That works but comes with some trade-offs: - You have to split the workload (e.g., multiple pipelines reading from the same source) - Transformation logic gets duplicated across pipelines - Stateful logic becomes harder to manage and keep consistent - Debugging and changes get more difficult because the data flow is fragmented Another challenge arises when working with high-cardinality keys like user IDs, session IDs, or request IDs, and when you need to handle longer time windows (24h or more). The state grows quickly and many systems rely on in-memory state, which makes it expensive and harder to recover from failures. We wanted to solve this problem and rebuild our approach at GlassFlow. Instead of scaling by adding more pipelines, we scale within a single pipeline by using replicas. Each replica consumes, processes, and writes independently, and the workload is distributed across them. In the benchmarks we’re sharing, this scales to 500k+ events/sec while still running stateful transformations and writing into ClickHouse. A few things we think are interesting: - Scaling is close to linear as you add replicas - Works with stateful transformations (not just stateless ingestion) - State is backed by a file-based KV store instead of relying purely on memory - The ClickHouse sink is optimized for batching to avoid small inserts - The product is built with Go Full write-up + benchmarks: https://www.glassflow.dev/blog/glassflow-now-scales-to-500k-... Repo: https://github.com/glassflow/clickhouse-etl Happy to answer questions about the design or trade-offs.
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 2d ago · opentrailpaper.com
Launched alongside, April 2026
the whole month →
- AG
Thought the resources for GPU arch were lacking, so here we are
Life & fun · Apr 2026 · jaso1024.com
- IB
Built a ~9M param LLM from scratch to understand how they actually work. Vanilla transformer, 60K synthetic conversations, ~130 lines of PyTorch. Trains in 5 min on a free Colab T4. The fish thinks the meaning of life is food. Fork it and swap the personality for your own character.
AI · Apr 2026 · github.com
- BC
Life & fun · Apr 2026 · sam-burns.com
- IB
With social media and now AI, its important to keep the indie web alive. There are many people who write frequently. Blogosphere tries to highlight them by fetching the recent posts from personal blogs across many categories. There are two versions: Minimal (HN-inspired, fast, static): https://text.blogosphere.app/ Non-minimal: https://blogosphere.app/ If you don't find your blog (or your favorite ones), please add them. I will review and approve it.
AI · Apr 2026 · text.blogosphere.app