I made an Open Source Personal Data Pipeline
Hi there! I've been working with data in one form or another, professionally, for about 5 years. I've been thinking about my own personal data and how it's used for at least twice that long. I've been sort of building something in my head for a while that solves my own problem and, in the beginning of this year, I found the opportunity to spend some time building it out. I'll leave the detailed explanation to the blog post but, in short, I built what amounts to an API crawler combined with a data processor to help you download your personal data from 3rd party services and work with it using…
What it does
In the maker’s words, at launch
Hi there! I've been working with data in one form or another, professionally, for about 5 years. I've been thinking about my own personal data and how it's used for at least twice that long. I've been sort of building something in my head for a while that solves my own problem and, in the beginning of this year, I found the opportunity to spend some time building it out. I'll leave the detailed explanation to the blog post but, in short, I built what amounts to an API crawler combined with a data processor to help you download your personal data from 3rd party services and work with it using relatively simple YAML recipes. It's using DuckDB under the hood, which is quite impressive at turning unstructured JSON into queryable form. The goal is making your personal cloud data available locally for backup, exploration, and combination. The tool works end-to-end for limited APIs and use cases right now. I'm working on solving a few of my own problems with it (events added to Obsidian daily notes, workout summaries across multiple trackers, mini-CRM linking events and emails to notes) and would be happy to help folks get this up and running. I have links to Discord and Substack (bottom of the post) if you want to lurk and see what happens and I welcome any and all contributions you are motivated to make! Thanks for checking it out!
Does the same job
all alternatives →- AIAscend.io – Smarter Data Pipelines2019 · ▲23
Hi HN, I’m Sean, the founder of Ascend.io (https://www.ascend.io). I’m really excited to post here and announce the launch of Ascend.io, a radical new way of designing, scaling, and automating data pipelines. Ascend is the result of nearly 4 years of development effort for a team that is now 30-strong, and I would love for you to give it a test drive and let me what you think. I’ve felt this pain since I wrote my first MapReduce in 2004 (using Sawzall @ Google), and in the 15 years since, things have not improved at the pace of other parts of the technology ecosystem. When I went…
- ISI started a repo for sharing algorithm implementations2013 · github.com · ▲52
Everything that would be here is in the README. I hope this gets big, it has tons of potential.
- DTDownload Twitter data without API keys2022 · tweetfeast.com · ▲57
In April last year I started thinking about using Twitter in a smarter way. I wanted to do analytics on my tweets and find out more about people following me on Twitter. What kinds of things do people who follow me like and retweet? I decided to dig into the data and find out. When I went to try and download Twitter data in raw form I found I quickly got bogged down writing API wrangling code and fiddling with API keys. I just wanted to crunch some data but here I was wrangling Twitter's API. This was such a frustrating experience it suddenly looked like an opportunity to me. Was there room…
- SLsoft-launching one of my pet projects2011 · ▲27
So a while ago I was thinking it would be neat to build a site that had the most absolute utility. The idea I came up with was a site to store those little protips that are super useful, if you know them ahead of time. I hired a guy to work on it with me, and ClueDB was born: http://cluedb.com/ Please give it a whirl (it uses Twitter auth as login but doesn't tweet anything) and tell me what you think! (Various trivia: My first try at a project coded by someone else and "product managed" by me. Runs on Flask + MongoDB. Also, this is totally unrelated to my startup.)
- DODataHub, open source datasets for Artificial Intelligence2017 · ▲14
Hi HN! I am an undergrad student trying to build interesting things with AI. Recently, I was looking for a dataset I could use for a new project. I realized that it is really frustrating to go through all the government websites (with terrible UX) just to find some usable dataset. I set out to build a GitHub for datasets, named DataHub. Right now, we have more than 1000 datasets from Montréal and New York City, with more cities coming soon (and possible government agencies). All of this is wrapped into a powerful search. It's a breeze to find a dataset to work on. I'd be interested to know…
- MWmy weekend project, understanding the Silk Road anonymous marketplace2011 · burntbrunch.github.com · ▲40
This was more of a sandbox to play with Raphael and Flot than anything else but I think there are some interesting statistics in there. It'd be awesome to do this over time but I really don't have the spare time required.
More dev tools this month
the category →
Open-source GTM skills for technical founders
Dev tools · 29d ago · gtmcofounder.com
OpenTrailPaper is open-source bike computer firmware for the LilyGO T5S3 4.7" E-Paper PRO. It supports offline maps, GPX routes, FIT recording and Bluetooth sensors.
Dev tools · 1d ago · opentrailpaper.com
Launched alongside, July 2024
the whole month →
- IC
Many years ago, I made VJ softwares (to mix live visuals in clubs) for unexpected platforms like the Game Boy Advance, the Playstation 2 and the Raspberry Pi. This year, I’m back with a new web-app: Pikimov. Inspired by Photopea (a free Photoshop clone), I created this web-based motion design & video editor as an alternative to After Effects, to fill empty void. It's free, without signup, without cloud uploads (your files stay on your machine), and your projects are not used for AI models training.
AI · 2024 · pikimov.com
