Dev tools · alternatives · 2026
24 alternatives to Connect Pub/Sub to BigQuery with an open source Dataflow pipeline
A highly configurable Google Cloud Dataflow pipeline that writes data into Google Big Query table from Pub/Sub - bomboradata/pubsub-to-bigquery
This is an open source Apache Beam pipeline that transfers messages from Google Cloud Pub/Sub to BigQuery. It is designed for developers who need to stream data between these Google Cloud services. The solution uses… Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes. Connect Pub/Sub to BigQuery with an open source Dataflow pipeline launched in 2017; newer entries below may have overtaken it.
- 1
Every data pipeline job I had to tackle required quite a few components to set up: - One tool to ingest data - Another one to transform it - If you wanted to run Python, set up an orchestrator - If you need to check the data, a data quality tool Let alone this being hard to set up and taking time, it is also pretty high-maintenance. I had to do a lot of infra work, and while this being billable hours for me I didn’t enjoy the work at all. For some parts of it, there were nice solutions like dbt, but in the end for an end-to-end workflow, it didn’t work. That’s why I decided to build an…
2024 · github.com · its alternatives →
- 2
Get and manipulate data with a visual programming interface.
2017 · pipes.digital · its alternatives →
- 3
[Deprecated] Load data from Kafka to any data warehouse. BQ sink is being supported in Firehose now. https://github.com/odpf/firehose - gojekfarm/beast
2019 · github.com · its alternatives →
- 4
An easy way to get all your interaction data in BigQuery
2017 · keen.io · its alternatives →
- 5IM
2017 · xyz.insightdataengineering.com · its alternatives →
- 6
An end-to-end GoodReads Data Pipeline for Building Data Lake, Data Warehouse and Analytics Platform. - san089/goodreads_etl_pipeline
2020 · github.com · its alternatives →
- 7
GitHub data on Google BigQuery and public datasets
2016 · changelog.com · its alternatives →
- 8OS
2019 · mprove.io · its alternatives →
- 9
TableFlow▲306Open-source CSV importer, alternative to Flatfile
2023 · tableflow.com · its alternatives →
- 10
Turn any website to API by several clicks (serverless and support SPA!) - t9tio/cloudquery
2019 · github.com · its alternatives →
- 11PC
Hi HN! Porter Cloud (https://porter.run/porter-cloud) is a Platform as a Service (PaaS) like Heroku, but we make it easy for you to migrate to AWS, Azure, or GCP when you're ready. Like Heroku, Porter takes care of a lot of generic DevOps work for you (like setting up CI/CD, containerizing your applications, autoscaling, SSL certificates, setting up a reverse proxy) and lets you deploy your apps with a few clicks — saving you a lot of time while developing. However, as you probably know, there’s a downside: platforms like this become constraining if and when your app…
2024 · its alternatives →
- 12
Move marketing data to BigQuery warehouse, no code required
2023 · whatagraph.com · its alternatives →
- 13
Hello Everyone! We built SQLFlow as a lightweight stream processing engine. We leverage DuckDB as the stream processing engine, which gives SQLFlow the ability to process 10's of thousands of messages a second using ~250MiB of memory! DuckDB also supports a rich ecosystem of sinks and connectors! https://sql-flow.com/docs/category/tutorials/ https://github.com/turbolytics/sql-flow We were tired of running JVM's for simple stream processing, and also of bespoke one off stream processors I would love your feedback, criticisms and/or…
Dec 2025 · sql-flow.com · its alternatives →
- 14
Hi HN! We are Ashish and Armend, founders of GlassFlow. We just launched our open-source streaming ETL that deduplicates and joins Kafka streams before ingesting them to ClickHouse https://github.com/glassflow/clickhouse-etl Why we built this: Dedup with batch data is straightforward. You load the data into a temporary table. Then, find only the latest versions of the record through hashes or keys and keep them. After that, move the clean data into your main table. But have you tried this with streaming data? Users of our prev product were running real-time analytics…
2025 · github.com · its alternatives →
- 15
2018 · console.cloud.google.com · its alternatives →
- 16
Steampipe▲69Open source SQL interface to your favorite cloud APIs🧑💻
2022 · steampipe.io · its alternatives →
- 17
- 18
A fast, fully managed, scalable NoSQL database service
2015 · cloud.google.com · its alternatives →
- 19PS
Hello HN, I am Sai Srirampur, one of the Co-founders of PeerDB. (https://github.com/PeerDB-io/peerdb). We spent the past 7 months building a solid experience to replicate data from Postgres to data warehouses. Now we're expanding to queues. PeerDB Streams provides a simple and native way to replicate changes as they happen in Postgres to Queues (Kafka, Redpanda, Google PubSub, etc). We use Postgres logical decoding to enable Change Data Capture (CDC). Blog post here: https://blog.peerdb.io/peerdb-streams-simple-native-postgres.... 10-min quickstart here:…
2024 · its alternatives →
- 20
Hi HN community, I’m Nick, co-founder/CEO of Transform.co. I’m thrilled to share MetricFlow, an open-source metric creation framework: https://github.com/transform-data/metricflow MetricFlow strives to make what has historically been an extremely repetitive process, writing SQL queries on core normalized data models, much more DRY. MetricFlow consolidates the definitions for joins, aggregations, filters, etc., and programmatically generates SQL to construct data marts. You can think of it like LookML, but more powerful and ergonomic (and open source!). The project…
2022 · github.com · its alternatives →
- 21
Flowpipe [1] is a new open source [2] cloud scripting engine from Turbot [3]. Some of you may already use Steampipe [4], our open-source [5] tool to live query cloud APIs. We believe DevOps teams need an easier way to orchestrate their cloud, build workflows and respond to events. But - because it's DevOps - that must be in code. In Flowpipe, pipelines are written in HCL and use simple primitives for HTTP requests, executing containers, running Lambda-compatible functions, querying databases like Steampipe and more. We have also published a large ecosystem of open source Flowpipe Mods [6].…
2023 · github.com · its alternatives →
- 22
A distributed, fast open-source graph database featuring horizontal scalability and high availability - GitHub - vesoft-inc/nebula: A distributed, fast open-source graph database featuring horizontal scalability and high availability
2020 · github.com · its alternatives →
- 23
- 24
Masthead Data▲122Know compute cost of every pipeline & model in your BigQuery
2023 · mastheadata.com · its alternatives →
Also compare
- I built an open-source data pipeline tool in Go alternatives
- Early-stage Yahoo Pipes spiritual successor alternatives
- Kafka to Google Big Query Persistence Service alternatives
- Stream to BigQuery by Keen IO alternatives
- Interactive map for architecting big data pipelines alternatives
- Goodreads Data Pipeline alternatives
Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →