Pathway – Build Mission Critical ETL and RAG in Python (NATO, F1 Used)
Hi HN data folks, I am excited to share Pathway, a Python data processing framework we built for ETL and RAG pipelines. https://github.com/pathwaycom/pathway We started Pathway to solve event processing for IoT and geospatial indexing. Think freight train operations in unmapped depots bringing key merchandise from China to Europe. This was not something we could use Flink or Elastic for. Then we added more connectors for streaming ETL (Kafka, Postgres CDC…), data indexing (yay vectors!), and LLM wrappers for RAG. Today Pathway provides a data indexing layer for live data…
Pathway – Build Mission Critical ETL and RAG in Python (NATO, F1 Used) launched on June 13, 2024 with 73 votes, #671 of 4,672 launches that month and more than 90% of that year's launches. The site was still online in September 2026, 2 years on.
In plain words
Pathway is a Python data processing framework for building ETL and RAG pipelines. It handles streaming data processing from sources like Kafka and Postgres, performs stateless and stateful transformations, and enables vector-based data indexing for retrieval-augmented generation. Built with a Rust runtime using Differential Dataflow for incremental computation, Pathway is designed for data engineers and developers who need to process live data updates and maintain in-memory pipelines at scale.
written from the facts on this page · September 2026
From the sources
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG. - pathwaycom/pathway
In the maker’s words, at launch
Hi HN data folks, I am excited to share Pathway, a Python data processing framework we built for ETL and RAG pipelines. https://github.com/pathwaycom/pathway We started Pathway to solve event processing for IoT and geospatial indexing. Think freight train operations in unmapped depots bringing key…
Does a similar job
all alternatives →
Build Live AI and RAG Pipelines in Minutes with YAML Templates2024 · pathway.com · ▲8Hello everyone! I am Jan, CTO and one of the creators of Pathway, the real-time data processing framework. I’m excited to share Pathway’s ready-to-use AI Pipelines, configurable with just YAML! These frameworks offer out-of-the-box solutions for AI search, RAG, and more—optimized for real-time indexing and in-memory processing. What makes it simple? YAML templates! The pipeline templates are fully customizable using YAMLs to fit your needs, from changing the data sources to the choice of the LLM model, all without touching Pathway’s Python code. Thanks to the Pathway data processing engine,…
I built an open-source data pipeline tool in Go2024 · github.com · ▲200Every data pipeline job I had to tackle required quite a few components to set up: - One tool to ingest data - Another one to transform it - If you wanted to run Python, set up an orchestrator - If you need to check the data, a data quality tool Let alone this being hard to set up and taking time, it is also pretty high-maintenance. I had to do a lot of infra work, and while this being billable hours for me I didn’t enjoy the work at all. For some parts of it, there were nice solutions like dbt, but in the end for an end-to-end workflow, it didn’t work. That’s why I decided to build an…
MetricFlow – open-source metric framework2022 · github.com · ▲98Hi HN community, I’m Nick, co-founder/CEO of Transform.co. I’m thrilled to share MetricFlow, an open-source metric creation framework: https://github.com/transform-data/metricflow MetricFlow strives to make what has historically been an extremely repetitive process, writing SQL queries on core normalized data models, much more DRY. MetricFlow consolidates the definitions for joins, aggregations, filters, etc., and programmatically generates SQL to construct data marts. You can think of it like LookML, but more powerful and ergonomic (and open source!). The project…
- SAShapedQL – A SQL engine for multi-stage ranking and RAGJan 2026 · playground.shaped.ai · ▲80
Hi HN, I’m Tullie, founder of Shaped. Previously, I was a researcher at Meta AI, worked on ranking for Instagram Reels, and was a contributor to PyTorch Lightning. We built ShapedQL because we noticed that while retrieval (finding 1,000 items) has been commoditized by vector DBs, ranking (finding the best 10 items) is still an infrastructure problem. To build a decent for you feed or a RAG system with long-term memory, you usually have to put together a vector DB (Pinecone/Milvus), a feature store (Redis), an inference service, and thousands of lines of Python to handle business logic…
Splitgraph - Build and share data with Postgres, inspired by Docker/Git2020 · splitgraph.com · ▲81EnterpriseDB (“EDB”), the leader in accelerating Postgres® in the enterprise, today announced its acquisition of Splitgraph, an early-stage disruptor that provides a Postgres compatible serverless SQL API for building data driven applications from hundreds of datasources. This move is seen as a mean
GlassFlow – OSS streaming dedup and joins from Kafka to ClickHouse2025 · github.com · ▲78Hi HN! We are Ashish and Armend, founders of GlassFlow. We just launched our open-source streaming ETL that deduplicates and joins Kafka streams before ingesting them to ClickHouse https://github.com/glassflow/clickhouse-etl Why we built this: Dedup with batch data is straightforward. You load the data into a temporary table. Then, find only the latest versions of the record through hashes or keys and keep them. After that, move the clean data into your main table. But have you tried this with streaming data? Users of our prev product were running real-time analytics…
More ai this month
the category →
Paintings by AI models: programs run through a simulation of oil paint.
AI · 22d ago · stillwet.art
FastRouter.ai▲364Route requests to the right LLM for cost, latency & quality
AI · 23d ago · fastrouter.ai
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, June 2024
the whole month →
La Growth Machine▲1,216Create personalized, multi-channel conversations at scale
AI · 2024 · lagrowthmachine.com
PyjamaHR▲1,132Hiring on autopilot. The AI applicant tracking system (ATS).
Work · 2024 · pyjamahr.com