You're totally right, I can't imagine giving access to my bank accounts either.
That's why I chose to manually enter the expenses.
Yes, you have to think about it every time, but I bet on the fact that it takes only a few seconds to add an expense in my app.
Unfortunately, I'm not able to find a better solution yet but maybe one day we'll find one !