Hacker News Reader

Comment by sksksk | Hacker News Reader

I run an personal AI agent locally (hermes), and use a combination of local models and whatever model I can get for a reasonable price (usually deepseek). After using it for a couple months, I'm converted. They're so good, the more you use it, the more useful it is; you internalise more of what it could be used for. It runs so many boring parts of my life, leaving me with more time to spend with my family. I liken it to release of the first iPhone; it didn't invent any new technology, but packaged different bits of tech together in a way that was consumer friendly. I think we'll look back at OpenClaw/Hermes/Muse/Instinct in the same way. I do worry about using purely cloud based offerings like Muse, I reckon there is a market for someone to package an consumer AI agent that lives on your home network.

View on HN

ok but what are the legitimate use cases you have for this? We were promised something similar with Google Home et al but they are glorified light switches and timers in my home. I run a homelab, I have technical ambitions, but I cannot think of a single thing in my life that I would rather talk to a local AI about than just do it myself or sit down and prompt Claude manually for.

That's fair, I'll give a couple of example of things that I've used it for...

I built an app (in codex), that goes through my emails, pulls out any receipts, and adds the purchases to a sqlite database.

I do all my grocery shopping online (through my supermarket), so our weekly shop is stored in the database.

Hermes uses that to predict what items will need restocking (based on how often I buy it, and when I last bought it).

Every week, it logs into the supermarket website on my behalf, books a delivery slot, and adds all the items that need restocking to the shop.

Throughout the week, if I run out of something, I take a photo of the barcode and send it to hermes, it adds it to the shop for me.

I built a similar app for travel; if I have any upcoming travel, hermes will send me updates (delays, platform changes etc...).

I do a lot of train travel for work, if my train is delayed, it will automatically apply for the compensation on my behalf.

It reads my emails, and pulls any actionable items out, and adds it to an Asana list. I can either do the tasks myself, or assign it to Hermes and it will do it for me.

As an example, I had an task to book swimming lessons for my son, hermes monitors the swimming lesson availability and automatically booked it when a suitable one becomes available.

Shopping is a good use case, I have a clock that needed a new battery, it wasn't a standard battery, and required a screwdriver to open the compartment. I took a photo of the clock, sent it to hermes, and it found out what battery took and ordered it of Amazon for me.

I wanted to buy some smart thermostats for our radiators, but wanted to be compatiable with our current smart heating system. Hermes is connected to my home assistant, so it figured out what heating system we have and found a compatiable set of eBay (this is slightly more involved than it sounds, as there is a new version of the heating system with a new set of completely incompatiable accesories).

> I took a photo of the clock, sent it to hermes,

How precisely do you do this? I'm interested in setting up a similar system, and I keep getting tripped up on how best to share information with the agent in a way that doesn't compromise security but is still easy to use.

Tailscale on a phone, on a server, and then whatever you and your agent come up with. I had a web app I built to talk to it. One day I wanted to share files and I just told it to implement that feature.

I don't use much off the shelf anymore. It's too easy to make my own hyper customized versions.

what app do you use on your phone to communicate with yoir server? just a terminal emulator?

Just ask it to build you one. The era of hyper-focused personal software is just beginning.

I built an orchestrator app with a nice voice interface so you can just talk to it about all your work in progress - which are sessions in herdr on your machines. That way you can do a lot just by talking to it through phone but when you're back at your desk you see it all in herdr. It's in beta tests with mobile webUI but iOS and Android should come very soon https://pitupitu.com

I use the WhatsApp bridge

Seems needlessly elaborate to me. A lot of this is a one shot prompt.

> We were promised something similar with Google Home et al but they are glorified light switches and timers in my home.

If only :-(

Ever since I switched them to Gemini, they're just crazy unreliable. For example, I have a routine to control multiple light switches at once. On some days, it happily executes the routine. In the remaining (about 50%) days, it says "I'm sorry - there is no routine called X. Would you like to run routine Y instead?" Why it can find routine Y but not X is beyond me.

So no, it can't even control smart devices any more.

Can you give some ideas of what you use it for? I'm amenable to the idea of automating some things I just don't know what I'm missing. Or using it for things I never thought of before.

I've given some examples in a sibling comment.

If you want to get started, I'd recommend just finding one thing to automate, and do what's necessary to get to that point. Once you've got that going, you can iterate.

For me, the thing I started with was the train tracking.

- In order to get train tracking working, it needed email access to get my trips.

- To get my email, I sync my email inboxes to a local IMAP server

- To check train times, it needs access to a train data API, I got an API key from realtimetrains.

- To apply for train delay compensation, it needs access to a browser, so I set up camoufox.

- It needs to log in to the compensation portal, so I set up a 1Password vault and gave it access to select credentials.

- To send me updates, it needs a gateway, so I set up a WhatsApp bridge.

So at this point, I've got an agent that can read my emails, browse the web, and chat to me over WhatsApp. Once you've got all that going, it becomes much easier to iterate and add new things.

isn't there so many things that the "consumer friendly" ai apps (e.g. muse, dots, duets, bots) offered by frontier ai labs cannot do, though?

for instance whenever i have them download a youtube video to transcribe subtitles from or fetch some text from a website verbatim, i get a "paraphrased" version citing copyright issues and they are pretty adamant / explicit about having me upload the materials (subtitles, archived website, etc.) myself to process, which defeats the purpose of using a consumer ai app. it seems like this cannot be avoided unless we use open source ai.

If there is something that the AI can't do, I don't use it for that.

Why not just use a script with yt-dlp and whisper?

downloading the entire audio just to run it through whisper would be an inefficient (and token costly) workaround, i think.

there are other cases beyond when the models thought my requests violate their mandates (e.g. copyright concerns), e.g. the lack of root access to fix configuration / certificate issues (for instance, when a browser within the app's environment has certificate issues that require root user access, but the "consumer friendly" ai app doesn't have a dev mode).

How do tokens and copyright things play into this? I don't really understand. You can just pipe the audio straight from yt-dlp into whisper and it just transcribes it directly.

sorry i didn't understand your question either. i didn't say tokens and copyright things in the same context?

re: copyright mandates of consumer ai apps, i was saying that they refuse to work with things they got from third-party sites (say, .vtt from youtube) verbatim and relay the output without somehow paraphrasing it, unless you directly feed them .vtt or somehow trick them into believing that you're the copyright owner.

re: token usage, i was saying that your suggested approach (yt-dlp and whisper) is more token costly (or at least compute heavy). say for a 2-hour youtube video, i'd need to run whisper on the audio, plus speaker diarization to identify who's saying what, instead of simply downloading the .vtt file directly.

How are you combining local models with hosted models? I have tried OpenClaw with some of the smaller local models (Qwen3.6-35B-A3B, Qwen3.8-Flash-Next) and found that while it can sometimes complete tasks, it would face plant often enough that it made it impossible to use. Do you have some tiered system where the hosted models dispatch to local models?

Rules, hooks, subagents. You have to set-up an orchestrator pattern with enforceable rules. Agentic Development Environments (ADE) help to visualize the pattern. Orca, SuperSet, etc... The orchestrator enforces the rules.

A write-up would be nice.

I tried OpenClaw in Jan/Feb, and while I saw the promise, in practice it kept breaking. Wasn't worth my time.

Have things improved much since then? Assuming I'll only use a cheap API model (no local)?

I'm always a bit wary about doing write up on things like this, as I kind of just muddle things together and definitely don't follow best practices.

I used OpenClaw a while back, I did find it useful, but it wasn't as reliable.

This time around, I found hermes a lot easier to use and configure. One thing I think has improved it is building apps to "pre-compute" data.

For example, with my email, instead of accessing my email servers directly, I have an IMAP server running locally; with mbsync running to download all the mail from all my email accounts locally. Then, when I ask hermes for something in my email, it's much quicker, doesn't hit any rate limits, and gets the full email (gmail sometimes doesn't send the full body of the email).

With my travel and purchase tracking. I parse an e-mail once as it arrive (I use a frontier model to generate a script which can parse e-mails from that merchant in the future). The parsed data is added to a SQLite database.

I ask hermes what's the address of the place we're staying at next week, It generates a SQL query, and queries the database.

Much cheaper/quicker/reliable than searching my e-mails for booking confirmations, and then parsing the e-mail.

My (unscientific) theory, is that doing things like that is much more important than the model or agent you use.

That's pretty great.

For me, the concern with Hermes, etc, is the security aspect. I was running OpenClaw in a dedicated VM and I carefully controlled what files it had access to. But I'd rather not go that route again.

The other major headache was it trying to configure itself, and then crashing.

Eventually I wrote my own bot that communicates with me on Telegram. I've manually enabled a few capabilities ("Add to my TODO list", "Add to my calendar", "Get weather" (in format I prefer)). The hope was I'd just add functionalities as I needed, and know how the whole flow works.

I didn't find value in things like soul.md, heartbeat, memory, etc (as in, they never really "worked"). I think most of the stuff you described are things I could easily add to my custom solution - I think that may still be a better approach than using hermes and wondering what the heck it's doing.

What do you use to allow the LLM to get content from web sites? I don't need it to do stuff - just simple navigation. The most complex thing would be "Here's my question - go to this Linux forum thread and navigate all the pages and see if anything there addresses my question."

For browsing, I use camoufox, it tries to hide the fact that it is an AI running a bot and seems to be fairly successful.

I set the browser window dimensions as the same is my phone, and added a VNC server. so if there are things I need to intervene on (like a captcha), I can log in and manually take control.

Thanks! I'll checkout camoufox.

How do you get notified that there is a captcha?

The agent will tell me in the WhatsApp chat, and send a screenshot.

I have a VNC client app on my phone, so I log in and take control. But my next step is to setup a web based VNC client, and have the agent send me a link to it (with time based with), so that I can do it from within WhatsApp

What are you doing to avoid sending your personal info to non-trusted "whatever models?" I wouldn't trust some of the cheaper models to have zero data retention, etc.

Reply on news.ycombinator.com