Melchi – Open-Source Snowflake to DuckDB Replication with CDC Support
Hey hacker news! I built Melchi, an open-source tool that handles Snowflake to DuckDB replication with proper CDC support. I'd love your feedback on the approach and potential use cases. *Why I built it:* When I worked at Redshift I saw two common scenarios that were painfully difficult to solve: Teams needed to query and join data from other organizations' Snowflake instances with their own data stored in different warehouse types, or they wanted to experiment with different warehouse technologies but the overhead of building and maintaining data pipelines was too high. With DuckDB's…
What it does
In the maker’s words, at launch
Hey hacker news! I built Melchi, an open-source tool that handles Snowflake to DuckDB replication with proper CDC support. I'd love your feedback on the approach and potential use cases. *Why I built it:* When I worked at Redshift I saw two common scenarios that were painfully difficult to solve: Teams needed to query and join data from other organizations' Snowflake instances with their own data stored in different warehouse types, or they wanted to experiment with different warehouse technologies but the overhead of building and maintaining data pipelines was too high. With DuckDB's growing popularity for local analytics, I built this to make warehouse-to-warehouse data movement simpler. *How it works:* * Supports three CDC strategies: standard streams (full change tracking), append-only streams (insert-only tables), and full refresh * Handles schema matching and type conversion automatically * Manages all the change tracking metadata * Uses DataFrames for efficient data movement instead of CSV dumps * Provides transactional consistency with automatic rollback on failures * Processes data in configurable batch sizes for memory efficiency Quick setup example: ```yaml source: type: snowflake account: ${SNOWFLAKE_ACCOUNT} warehouse: YOUR_WAREHOUSE change_tracking_schema: streams target: type: duckdb database: output/local.duckdb ``` *Current limitations:* * Geography/Geometry columns not supported with standard streams (Snowflake limitation) * Primary keys must be defined in Snowflake (or a row ID will be auto-generated) * All tables must be replaced together when modifying transfer configuration * Cannot replicate tables with identical schema/column names into DuckDB, even from different Snowflake databases *Questions for the community:* 1. What use cases do you see for this kind of tool? 2. What features would make this more useful for your workflow? 3. Any concerns about the approach to CDC? 4. What other source/target databases would be valuable to support? GitHub: https://github.com/ryanwith/melchi Discord: https://discord.gg/bTg9kJ92 Looking forward to your thoughts and feedback!
Does the same job
all alternatives →- SESnowflake Emulator – Local Snowflake Development with Go and DuckDBJan 2026 · github.com · ▲6
Hi HN! I built a Snowflake emulator for local development and testing. Testing Snowflake code locally is frustrating – you need a real account (expensive) or mock everything (tedious). I wanted something that just works with the standard [gosnowflake](https://github.com/snowflakedb/gosnowflake) driver or REST API. snowflake-emulator fixes this by: - Using DuckDB as the storage engine - Auto-translating Snowflake SQL (IFF→IF, NVL→COALESCE, DATEADD, etc.) - Supporting gosnowflake driver protocol – no code changes needed - Providing REST API v2 – use from any language…
- SVSirixDB – versioning through efficient snapshotting2018 · ▲55
I've already posted yesterday, but I'd really love to get comments, any kind of questions, suggestions and help would be greatly appreciated as it's an Open Source project of mine (and was for others during my studies at the University of Konstanz 6 years ago). Since then I spent countless ours to bring forth the idea of a versioned storage system, especially well suited for analytical tasks for timd-varying data. Especially I'd love to discuss what documentation you need, which next steps are necessary (JSON, Cloud...), API additions or changes... I've updated the README quiet a bit, such…
- ODOrchest – Data Science Pipelines2020 · ▲64
Hello Hacker News! We are Rick & Yannick from Orchest (https://www.orchest.io - https://github.com/orchest/orchest). We're building a visual pipeline tool for data scientists. The tool can be considered to be high-code because you write your own Python/R notebooks and scripts, but we manage the underlying infrastructure to make it 'just work™'. You can think of it as a simplified version of Kubeflow. We created Orchest to free data scientists from the tedious engineering related tasks of their job. Similar to how companies like Netflix, Uber and Booking.com…
- DRDuckDB + React = SQLRooms Framework for Analytics Apps Without Backend2025 · sqlrooms.org · ▲9
We’ve built SQLRooms, an open-source framework for creating single-node data analytics apps powered by DuckDB. It lets you build fully client-side, data-centric apps using React and DuckDB running in the browser (via WebAssembly) or in Electron. No server or backend is required — apps can work offline, preserve data privacy, and run queries on large datasets with sub-second performance. Features: - Query large datasets in browser with DuckDB (WASM) - Modular design for building composable data UIs (query editors, dashboards, notebooks, etc.) - Data privacy-preserving AI assistant that can…
- CECensus – Export for data warehouses2020 · ▲34
Hello Hacker News! We're a team of YC founders (Meldium W13, Draft S11, TapEngage S11) launching something new (https://www.getcensus.com). How many times has your business team asked you to generate yet another CSV file, write a ”quick report” in SQL, or send some custom data to a terrible API (looking at you Marketo)? We’ve built a product that connects directly to your data warehouse and syncs into apps like Salesforce, Customer.io and even Google Sheets. In fact, your business teams won’t even need to rely on engineering to manage all these pipelines. The tech stack for…
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 1d ago · opentrailpaper.com
Launched alongside, November 2024
the whole month →
- IB
I wasn't quite sure if this qualified as "Show HN" given you can't really download it and try it out. However, dang said[0]: > If it's hardware or something that's not so easy to try out over the internet, find a different way to show how it actually works—a video, for example, or a detailed post with photos. Hopefully I did that? Additionally, I've put code and a detailed guide for the netboot computer management setup on GitHub: https://github.com/kentonv/lanparty Anyway, if this shouldn't have been Show HN, I apologize! [0]…
Dev tools · 2024 · lanparty.house
