Arroyo – Write SQL on streaming data
Hey HN, Arroyo is a modern, open-source stream processing engine, that lets anyone write complex queries on event streams just by writing SQL—windowing, aggregating, and joining events with sub-second latency. Today data processing typically happens in batch data warehouses like BigQuery and Snowflake despite the fact that most of the data is coming in as streams. Data teams have to build complex orchestration systems to handle late-arriving data and job failures while trying to minimize latency. Stream processing offers an alternative approach, where the query is compiled into a streaming…
In plain words
Arroyo is an open-source stream processing engine that lets data teams write SQL queries on event streams to perform windowing, aggregating, and joining with sub-second latency. It compiles queries into streaming programs that continuously update as new data arrives, eliminating the need for complex batch orchestration systems. The tool is designed for data teams processing real-time event streams who want low-latency results without building custom infrastructure for handling late-arriving data and job failures.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
Hey HN, Arroyo is a modern, open-source stream processing engine, that lets anyone write complex queries on event streams just by writing SQL—windowing, aggregating, and joining events with sub-second latency. Today data processing typically happens in batch data warehouses like BigQuery and Snowflake despite the fact that most of the data is coming in as streams. Data teams have to build complex orchestration systems to handle late-arriving data and job failures while trying to minimize latency. Stream processing offers an alternative approach, where the query is compiled into a streaming program that constantly updates as new data comes in, providing low-latency results as soon as the data is available. I started the Arroyo project after spending the past five years building real-time platforms at Lyft and Splunk. I saw first hand how hard it is for users to build correct, reliable pipelines on top of existing systems like Flink and Spark Streaming, and how hard those pipelines are to operate for infra teams. I saw the need for a new system that would be easy enough for any data team to adopt, built on modern foundations and with the lessons of the past decade of research and industry development. Arroyo works by taking SQL queries and compiling them into an optimized streaming dataflow program, a distributed DAG of computation with nodes that read from sources (like Kafka), perform stateful computations, and eventually write results to sinks. That state is consistently snapshotted using a variation of the Chandy-Lamport checkpointing algorithm for fault-tolerance and to enable fast rescaling and updates of the pipelines. The entire system is easy to self-host on Kubernetes and Nomad. See it in action here: https://www.youtube.com/watch?v=X1Nv0gQy9TA or follow the getting started guide (https://doc.arroyo.dev/getting-started) to run it locally.
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 1d ago · opentrailpaper.com
Launched alongside, June 2023
the whole month →- NI
Heya HN, I've been working on a reddit-like platform as my primary side project for the last few years. Doing a (very) soft launch today, mainly because I want to use it to encourage discussion of alternatives. How non.io works: 1. Free to browse, paid to interact. 2. Minimum subscription is $2 (though you can choose more). I take $1 to run the servers, everything left gets split evenly between everything you upvote that month. It's a simple model, but I hope it's a better one than the freemium model we've been relying on for the last few years. Fundamentally I feel like any ad-supported…
Commerce · 2023 · non.io
Typefully 2.0▲931Effortlessly publish on Twitter and LinkedIn, now with AI
Growth · 2023 · typefully.com
SlashPage▲760Create websites in seconds to build your first 1K community
Growth · 2023 · slashpage.com
