Dev tools · alternatives · 2026

24 alternatives to Kafka to Google Big Query Persistence Service

[Deprecated] Load data from Kafka to any data warehouse. BQ sink is being supported in Firehose now. https://github.com/odpf/firehose - gojekfarm/beast

This service transfers data from Apache Kafka to Google BigQuery, enabling automated data pipeline integration between message streaming and cloud data warehousing. It is designed for teams managing real-time data… Below are 24 products that do a similar job, ranked by how close each is in meaning and then by launch-day votes. Prices are shown for the 1 we have checked on their own sites, 1 of them free or with a free tier. Kafka to Google Big Query Persistence Service launched in 2019; newer entries below may have overtaken it.

1. 1
   Connect Pub/Sub to BigQuery with an open source Dataflow pipeline▲44

   A highly configurable Google Cloud Dataflow pipeline that writes data into Google Big Query table from Pub/Sub - bomboradata/pubsub-to-bigquery

   2017 · github.com · its alternatives →

2. 2
   DuckDB for Kafka Stream Processing▲77

   Hello Everyone! We built SQLFlow as a lightweight stream processing engine. We leverage DuckDB as the stream processing engine, which gives SQLFlow the ability to process 10's of thousands of messages a second using ~250MiB of memory! DuckDB also supports a rich ecosystem of sinks and connectors! https://sql-flow.com/docs/category/tutorials/ https://github.com/turbolytics/sql-flow We were tired of running JVM's for simple stream processing, and also of bespoke one off stream processors I would love your feedback, criticisms and/or…

   Dec 2025 · sql-flow.com · its alternatives →

3. 3
   KQ – Simple Job Queue for Python Using Kafka▲116

   Kafka-based Job Queue for Python. Contribute to joowani/kq development by creating an account on GitHub.

   2016 · github.com · its alternatives →

4. 4PS
   Peerdb Streams – Simple, native Postgres change data capture▲177

   Hello HN, I am Sai Srirampur, one of the Co-founders of PeerDB. (https://github.com/PeerDB-io/peerdb). We spent the past 7 months building a solid experience to replicate data from Postgres to data warehouses. Now we're expanding to queues. PeerDB Streams provides a simple and native way to replicate changes as they happen in Postgres to Queues (Kafka, Redpanda, Google PubSub, etc). We use Postgres logical decoding to enable Change Data Capture (CDC). Blog post here: https://blog.peerdb.io/peerdb-streams-simple-native-postgres.... 10-min quickstart here:…

   2024 · its alternatives →

5. 5
   Whatagraph Data Transfer▲170

   Move marketing data to BigQuery warehouse, no code required

   2023 · whatagraph.com · its alternatives →

6. 6
   Google Cloud Bigtable▲99

   A fast, fully managed, scalable NoSQL database service

   2015 · cloud.google.com · its alternatives →

7. 7
   GlassFlow – OSS streaming dedup and joins from Kafka to ClickHouse▲78

   Hi HN! We are Ashish and Armend, founders of GlassFlow. We just launched our open-source streaming ETL that deduplicates and joins Kafka streams before ingesting them to ClickHouse https://github.com/glassflow/clickhouse-etl Why we built this: Dedup with batch data is straightforward. You load the data into a temporary table. Then, find only the latest versions of the record through hashes or keys and keep them. After that, move the clean data into your main table. But have you tried this with streaming data? Users of our prev product were running real-time analytics…

   2025 · github.com · its alternatives →

8. 8
   Streambed – Stream Postgres to Iceberg on S3, Supports Postgres Wire▲129

   Stream Postgres to Apache Iceberg on S3 via logical replication, queryable over the Postgres wire protocol. - GitHub - viggy28/streambed: Stream Postgres to Apache Iceberg on S3 via logical replication, queryable over the Postgres wire protocol.

   May 2026 · github.com · its alternatives →

9. 9
   Stream to BigQuery by Keen IO▲14

   An easy way to get all your interaction data in BigQuery

   2017 · keen.io · its alternatives →

10. 10
    Kafka 0.8.0 on Cloudflare Workers▲148

    Kafka 0.8.0 broker implementation on top of Cloudflare Workers - maxwellpeterson/kafka-worker

    2022 · github.com · its alternatives →

11. 11
    Goka – distributed stream processing library for Kafka written in Go▲105

    Goka is a compact yet powerful distributed stream processing library for Apache Kafka written in Go. - lovoo/goka

    2017 · github.com · its alternatives →

12. 12
    Golang service to read records from kafka and write to elasticsearch▲12

    Golang app to read records from a set of kafka topics and write them to an elasticsearch cluster - inloco/kafka-elasticsearch-injector

    2018 · github.com · its alternatives →

13. 13
    The Changelog - GitHub and Google on Public Datasets and BigQuery▲67

    GitHub data on Google BigQuery and public datasets

    2016 · changelog.com · its alternatives →

14. 14
    I simulated Bigtable in BigQuery as a Type 2 SCD▲15

    I hit a wall recently where we needed to design a data store that can both (1) ingest schema-less updates at scale and (2) allow for large analytical queries. GCP native doesn't have something that does both of these out-of-the-box, so we kinda hacked our own solution by replicating updates from Bigtable into a Type 2 SCD table in BigQuery. i couldn't find many resources online on this sort of trick when I was building it out, so I wrote up this blog with the details. Hope it's helpful and curious if anyone has used or built something similar. Will stick around for comments, happy to answer…

    2025 · statsig.com · its alternatives →

15. 15PD
    PicoMQ – Durable Streams over HTTP, on object storagefree▲159

    PicoMQ is a Rust server for Durable Streams, built on Object Store. Cheap, URL-addressable, granular streams (create/append/read/long-poll/SSE), with Pico Protocol or Durable Streams Protocol as the facade. S3Stream is the stream storage primitive, used in AutoMQ, shipped as a Rust library. Coordination is a command log in Postgres.

    Aug 2026 · picomq.com · its alternatives →

16. 16LV
    Live visualization of Kafka streams▲144

    2016 · alooma.com · its alternatives →

17. 17
    Execute SQL against Bluesky firehose▲100

    DuckDB for streaming data. Contribute to turbolytics/sql-flow development by creating an account on GitHub.

    2024 · github.com · its alternatives →

18. 18
    DataBackfill▲131

    Backfill your historical GA4 data to BigQuery

    2024 · databackfill.com · its alternatives →

19. 19
    Firebase2GraphQL▲106

    Migrate from Firebase to GraphQL on Postgres

    2018 · firebase2graphql.com · its alternatives →

20. 20
    Streambased▲114

    Analytics service for Kafka

    2024 · streambased.io · its alternatives →

21. 21
    AutoMQ - A Cost-Effective Kafka Distro That Can Autoscale in Seconds▲58

    Diskless Kafka® on S3. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency. Multi-AZ Availability. - AutoMQ/automq

    2024 · github.com · its alternatives →

22. 22
    UI for Apache Kafka▲158

    Open-Source Web UI for Apache Kafka Management. Contribute to provectus/kafka-ui development by creating an account on GitHub.

    2021 · github.com · its alternatives →

23. 23
    Firebase2graphql – Migrate from Firebase to GraphQL on Postgres▲114

    Blazing fast, instant realtime GraphQL APIs on all your data with fine grained access control, also trigger webhooks on database events. - graphql-engine/community/tools/firebase2graphql at master · hasura/graphql-engine

    2018 · github.com · its alternatives →

24. 24
    Fast Data Loader by SnapLogic▲71

    Easily move your data into cloud data warehouse, fast

    2021 · snaplogic.com · its alternatives →

Ranked by how close each launch is in meaning, then by votes. Prices were read from each product’s own site when checked and can change. Refine with your own description →