The only issues I've faced with this setup is one of codec availability and bitrate:
Sometimes, I'll be in a situation where I only have a high-quality, high bitrate copy of some media on my server, and I'll be trying to access it over a low bandwidth connection. If I'm stuck in one place for a while, I can re-encode the media on the server, but I'm unable to begin streaming to my device until the new lower bitrate copy is finished encoding on the server side.
Other times, I'll be in a situation where I want to access a file, but I need it in a different codec, so again I'll have to re-encode it on the server and download once that's finished.
What would be amazing, and would solve both problems (I'm not sure this is possible) is some kind of real time, http-based transcoding proxy. So you give it a link to a media file somewhere on the web, and it returns an http link which is streamed and transcoded in real time and served back to the client, all over http (so existing http media streaming clients can use this as a drop in replacement).
If such a thing existed (or exists) I'll donate $100 right now.