While it mainly worked out in the end, there were many stretches of years on end when I stuck it out at jobs that I didn't like or had outgrown, but were difficult to move from (or start my own company) because I was on a visa and could only move to some other company that would sponsor, or move internationally.
It makes other aspects of life harder too, e.g. starting a company, buying property - sure, you can buy, but do you really want to own somewhere if you might not be allowed to live/work there in the near future? Relationships - romantic/cultural issues, and you'll leave behind all your school friends, people you grew up with. Sure, you can make new friends... but they won't be as deep, and they'll probably kind of be other misfits in the new place, like you.
Not to say that it never works out... and maybe it's interesting for 2-3 years before going back home or settling some place permanently, but it can come at a pretty big cost. If moving from some third world place to a developed place, it's probably a no-brainer, but if you're moving from one developed country to another, it's quite possibly more paperwork and inconvenience than it's worth, and for the most part I wouldn't really recommend it, I think there is a lot to be said for just doing the best you can where you are, with your network, and with citizenship/no restrictions on anything.