Connect Pub/Sub to BigQuery with an open source Dataflow pipeline
A highly configurable Google Cloud Dataflow pipeline that writes data into Google Big Query table from Pub/Sub - bomboradata/pubsub-to-bigquery
Connect Pub/Sub to BigQuery with an open source Dataflow pipeline launched on August 9, 2017 with 44 votes, #788 of 3,214 launches that month and more than 90% of that year's launches. The site was still online in September 2026, 9 years on.
In plain words
This is an open source Apache Beam pipeline that transfers messages from Google Cloud Pub/Sub to BigQuery. It is designed for developers who need to stream data between these Google Cloud services. The solution uses Dataflow, Google's managed data processing service, to handle the message ingestion and delivery. It was first published on GitHub in 2017 and provides a ready-made template for this common data integration task.
written from the facts on this page · September 2026
Does a similar job
all alternatives →
I built an open-source data pipeline tool in Go2024 · github.com · ▲200Every data pipeline job I had to tackle required quite a few components to set up: - One tool to ingest data - Another one to transform it - If you wanted to run Python, set up an orchestrator - If you need to check the data, a data quality tool Let alone this being hard to set up and taking time, it is also pretty high-maintenance. I had to do a lot of infra work, and while this being billable hours for me I didn’t enjoy the work at all. For some parts of it, there were nice solutions like dbt, but in the end for an end-to-end workflow, it didn’t work. That’s why I decided to build an…
Early-stage Yahoo Pipes spiritual successor2017 · pipes.digital · ▲280Get and manipulate data with a visual programming interface.
Kafka to Google Big Query Persistence Service2019 · github.com · ▲11[Deprecated] Load data from Kafka to any data warehouse. BQ sink is being supported in Firehose now. https://github.com/odpf/firehose - gojekfarm/beast
Stream to BigQuery by Keen IO2017 · keen.io · ▲14An easy way to get all your interaction data in BigQuery
- IMInteractive map for architecting big data pipelines2017 · xyz.insightdataengineering.com · ▲144
The Changelog - GitHub and Google on Public Datasets and BigQuery2016 · changelog.com · ▲67GitHub data on Google BigQuery and public datasets
More dev tools this month
the category →
open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.
Dev tools · 27d ago · github.com
Contribute to reladraw/reladraw development by creating an account on GitHub.
Dev tools · 25d ago · github.com
OpenBot▲362Grok Bot alternative: free, local, open-source, multiplayer
Dev tools · 22d ago · openbot.run
Talanoa▲340Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…
Dev tools · 20d ago · talanoa.email
CoreSpeed▲332One MCP for everything your agents need: apps, memory, tools
Dev tools · 27d ago · corespeed.io
Source: Show HN post ↗
Launched alongside, August 2017
the whole month →
APIs to automatically transcribe and understand audio
AI · 2017 · assemblyai.com
The essential guide to product design and design thinking
Work · 2017 · designbetterpodcast.com
