Denormalized – Embeddable Stream Processing in Rust and DataFusion

In plain words

Denormalized is an embeddable stream processing engine built in Rust using Apache DataFusion. It enables incremental processing of large data volumes over time within a single application, following the efficiency model demonstrated by single-node systems like DuckDB. Designed for developers working with streaming workloads, it offers an alternative to distributed query engines by processing terabyte-scale data on a single node.

written from the facts on this page · September 2026

From the sources

Embeddable stream processing engine based on Apache DataFusion - probably-nothing-labs/denormalized

In the maker’s words, at launch

tl;dr we built an embeddable stream processing engine in Rust using apache DataFusion, check us out at https://github.com/probably-nothing-labs/denormalized Hey HN, We’d like to showcase a very early version of our embeddable stream processing engine called Denormalized. The rise of DuckDB has…

• 
  Python Stream Processing with Denormalized2024 · github.com · ▲20

  tl;dr we’re releasing python bindings for writing stream processing jobs using the Denormalized Rust library. Hey HN, A few months ago we released Denormalized, our embeddable stream processing engine written in rust [0]. Today we’re excited to showcase python bindings for the library. Python is still very much the language of data and we wanted to make it easy for more engineers to write fast streaming jobs without having to learn Rust. Currently, you can use the library to write streaming aggregations, stream joins, and user defined aggregate functions (thought the API for this is quite…

• 
  DuckDB for Kafka Stream ProcessingDec 2025 · sql-flow.com · ▲77

  Hello Everyone! We built SQLFlow as a lightweight stream processing engine. We leverage DuckDB as the stream processing engine, which gives SQLFlow the ability to process 10's of thousands of messages a second using ~250MiB of memory! DuckDB also supports a rich ecosystem of sinks and connectors! https://sql-flow.com/docs/category/tutorials/ https://github.com/turbolytics/sql-flow We were tired of running JVM's for simple stream processing, and also of bespoke one off stream processors I would love your feedback, criticisms and/or…

• 
  A fast HNSW implementation in Rust2024 · github.com · ▲111

  HNSW implementation in Rust. Reference: https://arxiv.org/ftp/arxiv/papers/1603/1603.09320.pdf - swapneel/hnsw-rust

• 
  SwimOS Rust – A framework for real-time streaming data applications2024 · github.com · ▲71
• AF
  Apache Fory Rust – 10-20x faster serialization than JSON/ProtobufOct 2025 · fory.apache.org · ▲67

  Serialization framework with some interesting numbers: 10-20x faster on nested objects than json/protobuf. Technical approach: compile-time codegen (no reflection), compact binary protocol with meta-packing, little-endian layout optimized for modern CPUs. Unique features that other fast serializers don't have: - Cross-language without IDL files (Rust ↔ Python/Java/Go) - Trait object serialization (Box) - Automatic circular reference handling - Schema evolution without coordination Happy to discuss design trade-offs. Benchmarks:…

• 
  All I wanted was to grok some real-time data2023 · teamdenormalized.substack.com · ▲8

  Hi HN, We are launching Denormalized (www.denormalized.io), a serverless real-time data platform built on Kafka and Pinot. We felt a bit burnt out by the sheer developer toil we faced when building application around the real-time data stack and set out to create a platform to allow small teams to be very productive with realtime data without having to glue together an elaborate system to serve real-time as well as time series queries. Here is our motivating post. Would appreciate any and all feedback.

• 
  /dev/null▲468

  a social network for devs

  Dev tools · 2d ago · dev-null.sh

• 
  Whiteboard (YC W26) – an open-source IDE for thoughtful software design▲423

  open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.

  Dev tools · 27d ago · github.com

• 
  Reladraw – A diagram language where you decide where to place things▲411

  Contribute to reladraw/reladraw development by creating an account on GitHub.

  Dev tools · 25d ago · github.com

• 
  OpenBot▲362

  Grok Bot alternative: free, local, open-source, multiplayer

  Dev tools · 22d ago · openbot.run

• 
  Talanoa▲340

  Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…

  Dev tools · 20d ago · talanoa.email

• 
  CoreSpeed▲332

  One MCP for everything your agents need: apps, memory, tools

  Dev tools · 27d ago · corespeed.io

• 
  Wordware▲9,848

  Your tool for building AI agents with natural language

  AI · 2024 · wordware.ai

• 
  If YouTube had actual channels▲2,741

  If Youtube had actual channels.

  Life & fun · 2024 · ytch.xyz

• 
  Me.bot▲1,627

  The inspiring companion for your life

  AI · 2024 · me.bot

• 
  Upmetrics AI▲1,197

  Business plan & financial forecast using AI

  Work · 2024 · upmetrics.co

• 
  Jupitrr AI▲1,195

  Jupitrr AI turns raw talking videos into scroll-stopping content for coaches and consultants. Research, script, record, edit, and publish in one app.

  AI · 2024 · jupitrr.com

• 
  InstantDB – A Modern Firebase▲1,145

  Hey there HN! We’re Joe and Stopa, and today we’re open sourcing InstantDB, a client-side database that makes it easy to build real-time and collaborative apps like Notion and Figma. Building modern apps these days involves a lot of schleps. For a basic CRUD app you need to spin up servers, wire up endpoints, integrate auth, add permissions, and then marshal data from the backend to the frontend and back again. If you want to deliver a buttery smooth user experience, you’ll need to add optimistic updates and rollbacks. We do these steps over and over for every feature we build, which can…

  Dev tools · 2024 · github.com