I built an open-source data pipeline tool in Go
Every data pipeline job I had to tackle required quite a few components to set up: - One tool to ingest data - Another one to transform it - If you wanted to run Python, set up an orchestrator - If you need to check the data, a data quality tool Let alone this being hard to set up and taking time, it is also pretty high-maintenance. I had to do a lot of infra work, and while this being billable hours for me I didn’t enjoy the work at all. For some parts of it, there were nice solutions like dbt, but in the end for an end-to-end workflow, it didn’t work. That’s why I decided to build an…
I built an open-source data pipeline tool in Go launched on December 17, 2024 with 200 votes, #191 of 5,365 launches that month and more than 95% of that year's launches. The site was still online in September 2026, 1 year on.
In plain words
Bruin is an open-source data pipeline framework built in Go that consolidates data ingestion, transformation, and Python orchestration into a single tool. Designed for data engineers tired of managing multiple separate components, it provides an end-to-end solution with a CLI application and includes built-in data quality checks. The tool aims to reduce infrastructure complexity and maintenance overhead compared to piecing together traditional point solutions.
written from the facts on this page · September 2026
From the sources
Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows. - bruin-data/bruin
In the maker’s words, at launch
Every data pipeline job I had to tackle required quite a few components to set up: - One tool to ingest data - Another one to transform it - If you wanted to run Python, set up an orchestrator - If you need to check the data, a data quality tool Let alone this being hard to set up and taking time, it is also pretty…
Does a similar job
all alternatives →
Open-source BI and analytics for engineers2024 · github.com · ▲293We are building Quary (https://quary.dev), an engineer-first BI/analytics product. You can find our repo at https://github.com/quarylabs/quary and our website at https://www.quary.dev/. There’s a demo video here: https://www.youtube.com/watch?v=o3hO65_lkGU As engineers who have worked on data at startups and Amazon, we were frustrated by self-serve BI tools. They seemed dumbed down and they always required us to abandon our local dev tools we know and love (e.g. copilot, git). For us and for everyone we speak to, they end up…
MetricFlow – open-source metric framework2022 · github.com · ▲98Hi HN community, I’m Nick, co-founder/CEO of Transform.co. I’m thrilled to share MetricFlow, an open-source metric creation framework: https://github.com/transform-data/metricflow MetricFlow strives to make what has historically been an extremely repetitive process, writing SQL queries on core normalized data models, much more DRY. MetricFlow consolidates the definitions for joins, aggregations, filters, etc., and programmatically generates SQL to construct data marts. You can think of it like LookML, but more powerful and ergonomic (and open source!). The project…
Open-source infra for building embedded data pipelines2022 · github.com · ▲41Hey HN! We are building *open source infrastructure for deploying customer-facing data pipelines.* Here’s our repo https://github.com/pipebird/pipebird and website https://pipebird.com/. Pipebird (YC W22) is designed to enable companies that generate important data to offer secure data pushes to their customers’ warehouses, directly from their products. Our team was previously building in fintech, where we heard from many of our peers that their customers wanted data pushed directly to their warehouses. Customers wanted to bring data into their source of…
SQLMesh – The Future of DataOps2023 · github.com · ▲29Hey Show HN! I’m Toby and over the last few months, I’ve been working with a team of engineers from Airbnb, Apple, Google, and Netflix, to simplify developing data pipelines with SQLMesh (https://github.com/TobikoData/sqlmesh). We’re tired of fragile pipelines, untested SQL queries, and expensive staging environments for data. Software engineers have reaped the benefits of DevOps through unit tests, continuous integration, and continuous deployment for years. We felt like it was time for data teams to have the same confidence and efficiency in development as their peers.…
Connect Pub/Sub to BigQuery with an open source Dataflow pipeline2017 · github.com · ▲44A highly configurable Google Cloud Dataflow pipeline that writes data into Google Big Query table from Pub/Sub - bomboradata/pubsub-to-bigquery
Prefect – A workflow engine designed for modern data engineering2019 · github.com · ▲59Prefect is a workflow orchestration framework for building resilient data pipelines in Python. - PrefectHQ/prefect
More dev tools this month
the category →
open-source IDE for thoughtful software design. Contribute to devdotfast/whiteboard development by creating an account on GitHub.
Dev tools · 27d ago · github.com
Contribute to reladraw/reladraw development by creating an account on GitHub.
Dev tools · 25d ago · github.com
OpenBot▲362Grok Bot alternative: free, local, open-source, multiplayer
Dev tools · 22d ago · openbot.run
Talanoa▲340Hi HN, I’m a fullstack developer, and honestly, I’ve lost count of how many side projects I’ve worked on over the years. But one idea kept coming back to me: a different kind of email client. It first came to me in 2014: “What if we could handle emails more like conversations?” Every year, I’d revisit this concept in my “Build Notes.” Ten years later, no one had built exactly what I had in mind — so I finally decided to do it myself. That’s how Talanoa was born. I wanted something focused on people, not timelines. I was tired of marking messages as unread just to remember to reply later, or…
Dev tools · 20d ago · talanoa.email
CoreSpeed▲332One MCP for everything your agents need: apps, memory, tools
Dev tools · 27d ago · corespeed.io
Source: Show HN post ↗ · launch text quoted in part
