Kafka to Google Big Query Persistence Service

In plain words

This service transfers data from Apache Kafka to Google BigQuery, enabling automated data pipeline integration between message streaming and cloud data warehousing. It is designed for teams managing real-time data flows who need to persist Kafka topics directly into BigQuery for analysis and storage. The tool handles the connection and data synchronization between these two platforms, making it useful for organizations already invested in both Kafka infrastructure and Google Cloud services.

written from the facts on this page · September 2026

• 
  Connect Pub/Sub to BigQuery with an open source Dataflow pipeline2017 · github.com · ▲44

  A highly configurable Google Cloud Dataflow pipeline that writes data into Google Big Query table from Pub/Sub - bomboradata/pubsub-to-bigquery

• 
  DuckDB for Kafka Stream ProcessingDec 2025 · sql-flow.com · ▲77

  Hello Everyone! We built SQLFlow as a lightweight stream processing engine. We leverage DuckDB as the stream processing engine, which gives SQLFlow the ability to process 10's of thousands of messages a second using ~250MiB of memory! DuckDB also supports a rich ecosystem of sinks and connectors! https://sql-flow.com/docs/category/tutorials/ https://github.com/turbolytics/sql-flow We were tired of running JVM's for simple stream processing, and also of bespoke one off stream processors I would love your feedback, criticisms and/or…

• 
  KQ – Simple Job Queue for Python Using Kafka2016 · github.com · ▲116

  Kafka-based Job Queue for Python. Contribute to joowani/kq development by creating an account on GitHub.

• PS
  Peerdb Streams – Simple, native Postgres change data capture2024 · ▲177

  Hello HN, I am Sai Srirampur, one of the Co-founders of PeerDB. (https://github.com/PeerDB-io/peerdb). We spent the past 7 months building a solid experience to replicate data from Postgres to data warehouses. Now we're expanding to queues. PeerDB Streams provides a simple and native way to replicate changes as they happen in Postgres to Queues (Kafka, Redpanda, Google PubSub, etc). We use Postgres logical decoding to enable Change Data Capture (CDC). Blog post here: https://blog.peerdb.io/peerdb-streams-simple-native-postgres.... 10-min quickstart here:…

• 
  Whatagraph Data Transfer2023 · whatagraph.com · ▲170

  Move marketing data to BigQuery warehouse, no code required

• 
  Google Cloud Bigtable2015 · cloud.google.com · ▲99

  A fast, fully managed, scalable NoSQL database service

• 
  Run full iOS 16.5 firmware on Android Devices▲513

  Contribute to oosexgang/hackios-xnu-kernel development by creating an account on GitHub.

  Dev tools · 1d ago · github.com

• 
  /dev/null▲492

  a social network for devs

  Dev tools · 4d ago · dev-null.sh

• 
  Whiteboard (YC W26) – an open-source IDE for thoughtful software design▲423

  open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.

  Dev tools · 29d ago · github.com

• 
  Reladraw – A diagram language where you decide where to place things▲411

  Contribute to reladraw/reladraw development by creating an account on GitHub.

  Dev tools · 27d ago · github.com

• 
  OpenBot▲362

  Grok Bot alternative: free, local, open-source, multiplayer

  Dev tools · 24d ago · openbot.run

• 
  Talanoa▲346

  Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…

  Dev tools · 22d ago · talanoa.email

• 
  Bel▲1,288

  Life & fun · 2019 · paulgraham.com

• 
  Vector Illustration Creator▲811

  Create free personalized illustrations from 3000+ elements

  Growth · 2019 · icons8.com

• 
  Open Doodles▲687

  A collection of free sketchy illustrations

  Growth · 2019 · opendoodles.com

• 
  AirPods Pro▲576

  New AirPods with active noise cancellation

  Life & fun · 2019 · apple.com

• 
  The New Ember Mugs▲568

  The next generation of temperature controlled mugs

  Life & fun · 2019 · ember.com

• 
  Lazy IoT▲533

  Control your environment with one easy keyboard shortcut

  Work · 2019 · lazy-app.com