Interesting! What makes it hard for you to develop trust? For me, I think that if I can't trust someone than I don't want them working for me in any capacity (office or remote!)
Not the OP, but I would imagine part of this comes from the fact that when you work side-by-side with someone, your level of non-verbal communication and just the ability to get to know one another is generally going to be a lot higher. You trust someone through getting to know them better. I'm not saying you can't achieve this level of communication, familiarity, and trust remotely, but I would assume it takes more planned effort to do so.
I guess this boils down to difficulties in building strong relationships remotely. I would definitely recommend traveling occasionally to build relationship with your remote folks.