Pathway – Build Mission Critical ETL and RAG in Python (NATO, F1 Used)
Hi HN data folks, I am excited to share Pathway, a Python data processing framework we built for ETL and RAG pipelines. https://github.com/pathwaycom/pathway We started Pathway to solve event processing for IoT and geospatial indexing. Think freight train operations in unmapped depots bringing key merchandise from China to Europe. This was not something we could use Flink or Elastic for. Then we added more connectors for streaming ETL (Kafka, Postgres CDC…), data indexing (yay vectors!), and LLM wrappers for RAG. Today Pathway provides a data indexing layer for live data…
In plain words
Pathway is a Python data processing framework for building ETL and RAG pipelines. It handles streaming data processing from sources like Kafka and Postgres, performs stateless and stateful transformations, and enables vector-based data indexing for retrieval-augmented generation. Built with a Rust runtime using Differential Dataflow for incremental computation, Pathway is designed for data engineers and developers who need to process live data updates and maintain in-memory pipelines at scale.
written from the facts on this page · September 2026
From the sources
In the maker’s words, at launch
Hi HN data folks, I am excited to share Pathway, a Python data processing framework we built for ETL and RAG pipelines. https://github.com/pathwaycom/pathway We started Pathway to solve event processing for IoT and geospatial indexing. Think freight train operations in unmapped depots bringing key merchandise from China to Europe. This was not something we could use Flink or Elastic for. Then we added more connectors for streaming ETL (Kafka, Postgres CDC…), data indexing (yay vectors!), and LLM wrappers for RAG. Today Pathway provides a data indexing layer for live data updates, stateless and stateful data transformations over streams, and retrieval of structured and unstructured data. Pathway ships with a Python API and a Rust runtime based on Differential Dataflow to perform incremental computation. All the pipeline is kept in memory and can be easily deployed with Docker and Kubernetes (pipelines-as-code). We built Pathway to support enterprises like F1 teams and NATO to build mission-critical data pipelines. We do this by putting security and performance first. For example, you can build and deploy self-hosted RAG pipelines with local LLM models and Pathway’s in-memory vector index, so no data ever leaves your infrastructure. Pathway connectors and transformations work with live data by default, so you can avoid expensive reprocessing and rely on fresh data. You can install Pathway with pip and Docker, and get started with templates and notebooks: https://pathway.com/developers/showcases We also host demo RAG pipelines implemented 100% in Pathway, feel free to interact with their API endpoints: https://pathway.com/solutions/rag-pipelines#try-it-out We'd love to hear what you think of Pathway!
Does the same job
all alternatives →- BLBuild Live AI and RAG Pipelines in Minutes with YAML Templates2024 · pathway.com · ▲8
Hello everyone! I am Jan, CTO and one of the creators of Pathway, the real-time data processing framework. I’m excited to share Pathway’s ready-to-use AI Pipelines, configurable with just YAML! These frameworks offer out-of-the-box solutions for AI search, RAG, and more—optimized for real-time indexing and in-memory processing. What makes it simple? YAML templates! The pipeline templates are fully customizable using YAMLs to fit your needs, from changing the data sources to the choice of the LLM model, all without touching Pathway’s Python code. Thanks to the Pathway data processing engine,…
- AWA working reference implementation of context engineeringApr 2026 · github.com · ▲46
I've been presenting at local meetups about Context Engineering, RAG, Skills, etc.. I even have a vbrownbag coming up on LinkedIn about this topic so I figured I would make a basic example that uses bedrock so I can use it in my talks or vbrownbags. Hopefully it's useful.
- AIAlerting in realtime RAG: spot changes to LLM answers, using few tokens2023 · github.com · ▲8
Hi I am Jan, CTO @ Pathway. A use case we have been working on with LLMs is to let people know when an answer to their query changes due to revisions of source documents. Obviously, we want to avoid periodically re-computing all queries for the LLM. Why I think it’s cool? - We don’t spin in a loop to repeat with the LLM. - Alerts are LLM-deduplicated - no spamming users with typo fixes - And the best - our framework, Pathway takes care of handling the updates, the example looks nearly like a regular, static RAG chatbot. More context + GIF of how it works for Google Drive document alerts:…
- PPPython package for generating accurate SQL via LLMs using RAG2023 · github.com · ▲7
Hello HN! We’ve been working hard on Vanna, our RAG framework for SQL generation and we’ve been updating our documentation. Please have a look — we have a ton of Jupyter notebooks for any combination of desired use cases. At it’s heart, we have abstractions that help you: - “train” a RAG “model” i.e. add metadata for the retrieval augmentation system to reference when constructing the LLM prompt (yes, we know that the terms “train” and “model” are somewhat confusing and we’re open to changing those terms if you can suggest better ones) - “ask” questions, which will generate SQL, run it,…
- SWSmartBuckets – With one line of code, never build a RAG pipeline again2025 · ▲5
Hey HN, We’re Basia, Fokke, and Geno from Liquidmetal AI, and we built something we wish we had a long time ago: SmartBuckets. We’ve spent a lot of time building RAG and AI systems, and honestly, the infrastructure side has always been a pain. Every project turned into a mess of vector databases, graph databases, and endless custom pipelines before you could even get to the AI part. SmartBuckets is our take on fixing that. It works like an object store, but under the hood it handles the messy stuff — vector search, graph relationships, metadata indexing — the kind of infrastructure you'd…
More ai this month
the category →
I trained a 125M-parameter transformer to autocomplete piano performances in real time (~108 notes/sec on an iPhone 15). The idea is basically GitHub Copilot or Tabnine, except instead of prompting it with code, you prompt it by playing a few notes on a MIDI piano. The model then continues what you played, entirely on-device. The app is free if anyone wants to try it. Happy to answer questions about the model, training, Core ML, or the many things that didn't work.
AI · 17d ago · simedw.com
Astute▲585Automate your B2B brand going viral, with new media creators
AI · 18d ago · company-app.joinastute.com
Hey HN, Henry from Cactus here! We previously released Cactus Needle, a 14MB agentic LLM for tool call, device use, and structured extraction for phones, wearables, smart homes, small robots and microcontrollers. We got really great feedback here, and have now incorporated the suggestions to release Needle 2. The whole model is a single 14MB binary that runs a full session in 28MB of RAM; 45m parameters at 2bit compression. Needle hits 500 tokens/sec decode speed on a Raspberry Pi 5, sits between 400-1,500 tokens/sec on VR devices like Meta Quest 3S and Apple Vision Pro, and ranges…
AI · 27d ago · cactuscompute.com
Launched alongside, June 2024
the whole month →
La Growth Machine▲1,216Create personalized, multi-channel conversations at scale
AI · 2024 · lagrowthmachine.com
PyjamaHR▲1,132Hiring on autopilot. The AI applicant tracking system (ATS).
Work · 2024 · pyjamahr.com