DataForge Core – Pure functional SQL data transforms
Hey HN, A few friends and I have spent the past 6 years or so developing a way to write data transformation code in a way that can easily adapt to changes in logic or data elements both upstream and downstream without the need major refactoring, regression testing, or re-orchestration. We decided to open source the project about two months ago and published a CLI tool after we realized how big of a task it was to take on incumbents like stored procedures, dbt, and psyspark. It is early days for our community and we are looking to grow and engage with others to poke holes and contribute…
What it does
In the maker’s words, at launch
Hey HN, A few friends and I have spent the past 6 years or so developing a way to write data transformation code in a way that can easily adapt to changes in logic or data elements both upstream and downstream without the need major refactoring, regression testing, or re-orchestration. We decided to open source the project about two months ago and published a CLI tool after we realized how big of a task it was to take on incumbents like stored procedures, dbt, and psyspark. It is early days for our community and we are looking to grow and engage with others to poke holes and contribute ideas! For an overview of the concepts and why we built it, our most recent blog does a decent job introducing the core ideas: https://www.dataforgelabs.com/blog/introduction-dataforge-fr... We will be publishing more content in the near future to dig into how we actually took these ideas and implemented them across the project. Feel free to submit ideas or issues directly to the Github!
Does the same job
all alternatives →- MOMetricFlow – open-source metric framework2022 · github.com · ▲98
Hi HN community, I’m Nick, co-founder/CEO of Transform.co. I’m thrilled to share MetricFlow, an open-source metric creation framework: https://github.com/transform-data/metricflow MetricFlow strives to make what has historically been an extremely repetitive process, writing SQL queries on core normalized data models, much more DRY. MetricFlow consolidates the definitions for joins, aggregations, filters, etc., and programmatically generates SQL to construct data marts. You can think of it like LookML, but more powerful and ergonomic (and open source!). The project…
- PIParabola.io – Automate your work with visual programming2018 · parabola.io · ▲291
Hi HN! I’m Alex from Parabola (https://parabola.io). Parabola is a visual programming tool for creating functional data flows that everyone can use. It’s entirely drag-and-drop, handles data sizes much larger than a traditional spreadsheet, calculates everything live, and can run your flows on a schedule of your choosing. I used to work in strategy consulting, doing data analytics for SMBs and Fortune 500 companies. The amount of time wasted on menial tasks was astounding. Things like cleaning data, generating custom reports, creating human workflows to solve shortcomings in third…
- BBBaseten – Build ML-powered applications2022 · baseten.co · ▲112
Hi HN, Three months ago, I took a job at Baseten to help craft and document an application builder that lets data scientists build full-stack, production-ready applications around their ML models without worrying about containers, Flask, or React. From my first day, everyone was focused on what would happen today: opening up our public beta. I’m super excited to see what you build with Baseten. If you want to take Baseten for a full-speed test drive, follow along with this tutorial, where you can build and deploy an application in 20 minutes:…
- DSDBML – simple DSL language to document database schemas2019 · dbml-lang.org · ▲57
- DMDB Migrations as Clojure data2014 · github.com · ▲31
- WMWe made Trellis – a way to run SQL query on your unstructured data2024 · demo.runtrellis.com · ▲8
Hey HN — We're excited to share Trellis — a snowflake for unstructured data. We've built an AI engine that turns unstructured data into structured SQL-format based on the schema you define in natural language. We spent a lot of time building ML infrastructure and realized that most data warehouses and data pipelines are not designed for unstructured data (documents, PDFs, calls). While something like a Vector database and RAG are great at search tasks, they really struggle with aggregation and SQL type queries such as 1. How many emails in the past 6 months contain complaints about the…
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 1d ago · opentrailpaper.com
Launched alongside, July 2024
the whole month →
- IC
Many years ago, I made VJ softwares (to mix live visuals in clubs) for unexpected platforms like the Game Boy Advance, the Playstation 2 and the Raspberry Pi. This year, I’m back with a new web-app: Pikimov. Inspired by Photopea (a free Photoshop clone), I created this web-based motion design & video editor as an alternative to After Effects, to fill empty void. It's free, without signup, without cloud uploads (your files stay on your machine), and your projects are not used for AI models training.
AI · 2024 · pikimov.com
