Streaming DataFrames–a Pandas-like syntax for real-time data
Hey all! We’ve built a Pandas-like interface to make it easy to work with streaming data using what we call ‘Streaming DataFrames’. For example, suppose that you want to convert speed measurement units from meters per second to kilometers per hour With static data in Pandas, you’d do this: df["speed_km_h"] = df["speed_m_s"] * 3.6 With Streaming DataFrames, it’s pretty much the same thing… sdf["speed_km_h"] = sdf["speed_m_s"] * 3.6 …except it’s being done continuously and the updated records can be sent to an output topic in Kafka with almost no delay after they’ve been processed. You can…
What it does
In the maker’s words, at launch
Hey all! We’ve built a Pandas-like interface to make it easy to work with streaming data using what we call ‘Streaming DataFrames’. For example, suppose that you want to convert speed measurement units from meters per second to kilometers per hour With static data in Pandas, you’d do this: df["speed_km_h"] = df["speed_m_s"] * 3.6 With Streaming DataFrames, it’s pretty much the same thing… sdf["speed_km_h"] = sdf["speed_m_s"] * 3.6 …except it’s being done continuously and the updated records can be sent to an output topic in Kafka with almost no delay after they’ve been processed. You can also do a stateful tumbling window like this: sdf = sdf.apply(lambda row: row["speed_km_h"]).tumbling_window(timedelta(seconds=3O), grace_ms=timedelta(seconds=1).mean().final() I know some peeps might be thinking “eeeeww Pandas”, but we wanted to make it more user-friendly for data and ML folks who are interested in moving batch to real-time or near real-time processing. Of course there’s Flink and Spark which are great stream processing tools… but if you need the specialised talent to operate them. We think there’s a need for something that's easier for generalists to learn, which is why it’s strictly Python only. It doesn’t wrap any other technology so you don’t need to do any cross language debugging or include SQL statements in your Python code. If you want to learn more about how it works, check out our launch blog: https://quix.io/blog/introducing-streaming-dataframes Github repo is here: https://github.com/quixio/quix-streams
Does the same job
all alternatives →- RTReal Time Stream Processing Startup2020 · ▲6
We are working on a product which makes stream processing easy for data analysts and data scientists. It’s a higher level alternative to Flink, Kafka Streams and Spark Streams. As well as making stream processing simpler, we also felt that lots of people were building these AWS stream processing platforms based on Kafka and something like Flink so wanted to offer this as a service. We would welcome any input on the space or approach: Home page: http://Timeflow.systems Demo: https://youtu.be/fGfgBaHXOkE
- BABamboolib – A GUI for Pandas (Python Data Science)2019 · bamboolib.com · ▲119
- RTReal Time Analytics SaaS2021 · ▲7
We have been involved in a number of real time streaming projects using tools such as Flink, Spark Streams and Kafka Streams backed by "real time" databases such as Druid. We always found these projects quite complex to develop and run, with stream processing in particular being a bit of a dark art. A stream to stream join in Flink can get quite mind bending for instance. In 2020 we had the idea of building a low code SaaS product for real time streaming analytics. The first attempt failed due to being a little over-engineered and with too many changes of direction, but over the last few…
- SPSelf-Published Book – “Data Science in Production”2020 · ▲172
Hi HN, Over the past 6 months I've been working on a technical book focused on helping aspiring data scientists to get hands-on experience with cloud computing environments using the Python ecosystem. The book is targeted at readers already familiar with libraries such as Pandas and scikit-learn that are looking to build out a portfolio of applied projects. To author the book, I used the Leanpub platform to provide drafts of the text as I completed each chapter. To typeset the book, I used the R bookdown package by Yihui Xie to translate my markdown into a PDF format. I also used Google docs…
- PIPandas in JS2017 · stratodem.github.io · ▲14
More life & fun this month
the category →- TL
Life & fun · 10d ago · louisabraham.github.io
Photosynthesis fires two of your iPhone
Life & fun · 28d ago · photosynthesis.camera
SoloUno▲310Take control of hair pulling, nail biting & skin picking
Life & fun · 28d ago · solouno.io
Scroll through all 43,252,003,274,489,856,000 reachable Rubik's Cube permutations.
Life & fun · 26d ago · everycube.alen.is
Hi HN, I built Eigendrum, a web tool that solves the 2D wave equation for arbitrary shapes so you can hear what they sound like as drums. How it works: * Solves -∇²u = λu using finite element analysis (Kφ = λMφ) on a triangle mesh. * Validated to <0.1% error against closed-form solutions for circles (Bessel zeros) and rectangles. * Sound model factors in strike location, Rayleigh damping, and mallet width. * Includes Kac drums I & II to demonstrate identical sound spectra from different geometries. * No frameworks, build steps, or dependencies. Repo and tests:…
Life & fun · 26d ago · baselashraf81.github.io
Launched alongside, April 2024
the whole month →
Supabase▲2,328The Postgres developer platform is now generally available
Dev tools · 2024 · supabase.com
Cal.com Platform▲1,602Build your pixel-perfect booking experience with Atoms
Dev tools · 2024 · cal.com
deco.cx 2.0▲1,080Build web apps 10x faster with Deno, JSX, TS & Tailwind
Dev tools · 2024 · decocms.com
