nowfound

Commerce · May 27, 2025

IS

I simulated Bigtable in BigQuery as a Type 2 SCD

I hit a wall recently where we needed to design a data store that can both (1) ingest schema-less updates at scale and (2) allow for large analytical queries. GCP native doesn't have something that does both of these out-of-the-box, so we kinda hacked our own solution by replicating updates from Bigtable into a Type 2 SCD table in BigQuery. i couldn't find many resources online on this sort of trick when I was building it out, so I wrote up this blog with the details. Hope it's helpful and curious if anyone has used or built something similar. Will stick around for comments, happy to answer…

Visit statsig.com ↗Alternatives

In plain words

This is a technical guide documenting how to replicate Bigtable data into BigQuery using Type 2 Slowly Changing Dimensions to handle both schema-less data ingestion at scale and large analytical queries. It's for engineers working with Google Cloud Platform who need a hybrid approach combining NoSQL flexibility with SQL analytics capabilities. The solution addresses a gap in GCP's native tools by providing a practical implementation pattern with detailed documentation.

written from the facts on this page · September 2026

From the sources

In the maker’s words, at launch

I hit a wall recently where we needed to design a data store that can both (1) ingest schema-less updates at scale and (2) allow for large analytical queries. GCP native doesn't have something that does both of these out-of-the-box, so we kinda hacked our own solution by replicating updates from Bigtable into a Type 2 SCD table in BigQuery. i couldn't find many resources online on this sort of trick when I was building it out, so I wrote up this blog with the details. Hope it's helpful and curious if anyone has used or built something similar. Will stick around for comments, happy to answer any questions or feedback.

More commerce this month

the category →
  • PaymentKit▲466

    Billing that survives a processor shutdown

    Commerce · 13d ago · paymentkit.com

  • Equitybee Benchmark▲241

    Compare your startup equity grant for free.

    Commerce · 26d ago · equitybee.com

Launched alongside, May 2025

the whole month →