I simulated Bigtable in BigQuery as a Type 2 SCD
I hit a wall recently where we needed to design a data store that can both (1) ingest schema-less updates at scale and (2) allow for large analytical queries. GCP native doesn't have something that does both of these out-of-the-box, so we kinda hacked our own solution by replicating updates from Bigtable into a Type 2 SCD table in BigQuery. i couldn't find many resources online on this sort of trick when I was building it out, so I wrote up this blog with the details. Hope it's helpful and curious if anyone has used or built something similar. Will stick around for comments, happy to answer…
I simulated Bigtable in BigQuery as a Type 2 SCD launched on May 27, 2025 with 15 votes, #1,017 of 7,905 launches that month and more than 75% of that year's launches. The site was still online in September 2026, 1 year on.
In plain words
This is a technical guide documenting how to replicate Bigtable data into BigQuery using Type 2 Slowly Changing Dimensions to handle both schema-less data ingestion at scale and large analytical queries. It's for engineers working with Google Cloud Platform who need a hybrid approach combining NoSQL flexibility with SQL analytics capabilities. The solution addresses a gap in GCP's native tools by providing a practical implementation pattern with detailed documentation.
written from the facts on this page · September 2026
From the sources
How we bent BigQuery to behave like Bigtable with Type 2 SCD modeling to enable querying schema-less data at scale in GCP native.
In the maker’s words, at launch
I hit a wall recently where we needed to design a data store that can both (1) ingest schema-less updates at scale and (2) allow for large analytical queries. GCP native doesn't have something that does both of these out-of-the-box, so we kinda hacked our own solution by replicating updates from Bigtable into a Type…
Does a similar job
all alternatives →
Google Cloud Bigtable2015 · cloud.google.com · ▲99A fast, fully managed, scalable NoSQL database service
- OSOpen-Source Business Intelligence for BigQuery – Looker Alternative2019 · mprove.io · ▲95
Connect Pub/Sub to BigQuery with an open source Dataflow pipeline2017 · github.com · ▲44A highly configurable Google Cloud Dataflow pipeline that writes data into Google Big Query table from Pub/Sub - bomboradata/pubsub-to-bigquery
- DDData Diff – compare tables of any size across databases2022 · ▲127
Gleb, Alex, Erez and Simon here – we are building an open-source tool for comparing data within and across databases at any scale. The repo is at https://github.com/datafold/data-diff, and our home page is https://datafold.com/. As a company, Datafold builds tools for data engineers to automate the most tedious and error-prone tasks falling through the cracks of the modern data stack, such as data testing and lineage. We launched two years ago with a tool for regression-testing changes to ETL code…
PostgreSQL performance and cost across 23 EC2 instance typesJul 2026 · postgres.saneengineer.com · ▲97Hey! I'm Andrei. I got frustrated by how people tend to build overcomplicated backend systems, being "motivated" by big tech case studies and popular books. So, I started exploring lean architecture, and building my digital garden of ideas, approaches and data that align with this direction. Here I want to present one of the tools – Sizing tool for PostgreSQL. I've benchmarked PostgreSQL on different EC2 instances and disks, with different initial data sets to see performance that these instances can give you. And I've built a tool to visualize this data, which I welcome you to explore. So,…
GoSQL – A query engine in 319 LoC2024 · github.com · ▲81I've always been curious about how SQL engines actually work. So I built a minimum viable SQL engine in Go. - Supports CSV files as tables - Supports SELECT, FROM, WHERE, LIMIT It's very simple: 1. Parses query string 2. Converts it into an AST representation 3. Executes the query against the CSV 4. Returns the results
Source: Show HN post ↗ · launch text quoted in part
Launched alongside, May 2025
the whole month →
Tapflow 2.0▲1,125We built Tapflow for devs, designers, PMs, and tech folks who’ve been sitting on internal docs, templates, and systems — but never productized them. Upload a PDF, notes, or workshop deck, and Tapflow drafts a structured product (with chapters, teaser and pricing plans). It’s good for AI kits, onboarding guides, design systems, and frameworks. You can sell through Stripe, LemonSqueezy, or PayPal. Real-time collaboration is in early beta. No site builder, no LMS — clean design and just focused tools to ship. Fully bootstrapped and just launched on PH today →…
Work · 2025 · tapflow.ai
- C9
Life & fun · 2025 · felixrieseberg.github.io
Appwrite Sites▲990Appwrite 2.0 brings a new engine, a rebuilt Console, five database types, an S3 API, an OAuth 2.1 server, and organization-level Domains and Firewall.
Dev tools · 2025 · appwrite.io
