There are cases where this could be true. I could see, say, asking it to do some research for you or something. Or maybe grab some restaurant recommendations. Really anything that brings you options that you can then make a decision on. If I do end up using this tech, thats how I would feel comfortable using it.
When it comes to things like making purchases and such on your behalf, thats where I disagree. Even with boundaries it's just not deterministic enough for most peoples risk appetite IMO. And when I say "most people" I am not talking about HN folks.
And just to be clear, I do use this technology. I just am very aware that its not this magic bullet that the AI bros want us to believe it is. I used it this morning in fact to help me write up some code I felt too lazy to deal with. It very quickly and efficiently wrote what I asked. I was pleasantly surprised and _almost_ missed the very bad bug that it dropped right into the middle of it all. I'm not giving my credit card to that lol.