Can you give some ideas of what you use it for? I'm amenable to the idea of automating some things I just don't know what I'm missing. Or using it for things I never thought of before.
If you want to get started, I'd recommend just finding one thing to automate, and do what's necessary to get to that point. Once you've got that going, you can iterate.
For me, the thing I started with was the train tracking.
- In order to get train tracking working, it needed email access to get my trips.
- To get my email, I sync my email inboxes to a local IMAP server
- To check train times, it needs access to a train data API, I got an API key from realtimetrains.
- To apply for train delay compensation, it needs access to a browser, so I set up camoufox.
- It needs to log in to the compensation portal, so I set up a 1Password vault and gave it access to select credentials.
- To send me updates, it needs a gateway, so I set up a WhatsApp bridge.
So at this point, I've got an agent that can read my emails, browse the web, and chat to me over WhatsApp. Once you've got all that going, it becomes much easier to iterate and add new things.
for instance whenever i have them download a youtube video to transcribe subtitles from or fetch some text from a website verbatim, i get a "paraphrased" version citing copyright issues and they are pretty adamant / explicit about having me upload the materials (subtitles, archived website, etc.) myself to process, which defeats the purpose of using a consumer ai app. it seems like this cannot be avoided unless we use open source ai.
there are other cases beyond when the models thought my requests violate their mandates (e.g. copyright concerns), e.g. the lack of root access to fix configuration / certificate issues (for instance, when a browser within the app's environment has certificate issues that require root user access, but the "consumer friendly" ai app doesn't have a dev mode).
re: copyright mandates of consumer ai apps, i was saying that they refuse to work with things they got from third-party sites (say, .vtt from youtube) verbatim and relay the output without somehow paraphrasing it, unless you directly feed them .vtt or somehow trick them into believing that you're the copyright owner.
re: token usage, i was saying that your suggested approach (yt-dlp and whisper) is more token costly (or at least compute heavy). say for a 2-hour youtube video, i'd need to run whisper on the audio, plus speaker diarization to identify who's saying what, instead of simply downloading the .vtt file directly.