Show HN: I made an Open Source Personal Data Pipeline

Hi there!

I've been working with data in one form or another, professionally, for about 5 years. I've been thinking about my own personal data and how it's used for at least twice that long. I've been sort of building something in my head for a while that solves my own problem and, in the beginning of this year, I found the opportunity to spend some time building it out.

I'll leave the detailed explanation to the blog post but, in short, I built what amounts to an API crawler combined with a data processor to help you download your personal data from 3rd party services and work with it using relatively simple YAML recipes. It's using DuckDB under the hood, which is quite impressive at turning unstructured JSON into queryable form. The goal is making your personal cloud data available locally for backup, exploration, and combination.

The tool works end-to-end for limited APIs and use cases right now. I'm working on solving a few of my own problems with it (events added to Obsidian daily notes, workout summaries across multiple trackers, mini-CRM linking events and emails to notes) and would be happy to help folks get this up and running. I have links to Discord and Substack (bottom of the post) if you want to lurk and see what happens and I welcome any and all contributions you are motivated to make!

Thanks for checking it out!