Another issue is team building / comradery. I don't know my current coworkers nearly as well as I've been able to know coworkers from previous companies I've worked with. When remote you don't get to go to lunch, grab coffee or a beer with your coworkers. We do travel a lot to meet up, but I still only see people I work with closely when doing so. As the company grows it's much more difficult to even begin to meet everyone in the company which is much easier when in and office because everyday you can bump into someone different.
We've tried to solve this to some extent through randomly pairing two people every two weeks to schedule a short 15 minute meeting. However the scheduling itself is not automated and it's not always easy to schedule time due to lack of proper calendaring software and distributed share calendars. This is definitely something that could be solved but it's just another challenge of working remotely.
Call-centers are another industry that can be entirely remote.