It does seem that was the case and then if you can simply say "please never disappear for the whole day without telling me" and that would be the end of it. I think there's more going on here.
When you can turn to your coworker next to you and ask a simple one minute question that unblocks you, but you can't reach your remote coworker for an entire day and you thus fail to make progress for an entire day because of their truancy, you begin to resent them really quickly. It doesn't matter if you're their manager or not; it's super irritating.
Part of what makes corporate America suck is the kind of passive aggressive resentments that never get expressed. Suddenly the remote worker finds that everyone hates him and has no clue why.