For arranged meetings including the remote workers won't be a problem but there are many 'meetings' that happen in a work context where important information is exchanged that are informal that remote workers will miss out on.
What if George had been away getting coffee or at the bathroom when you started the conversation, he would have missed out anyway.