Yes, it can be a pain. Sometimes you have a 6am meeting and sometimes an 11pm meeting and sometimes both in the same day. You have to have your shit together enough to deal with day-long lag for non-urgent stuff, and you have to care enough to jump on urgent stuff at all hours.
Even if you might rather have a chummy relationship with your colleagues, you have to be a bit more distanced and professional in order to accommodate the time difference.
But if you have a good team, you also should ask yourself: is the convenience of collocation worth losing the team?
At the risk of veering OT: there’s a pretty good business in arbitrage around these cultural blockers. For instance there are thousands of people working for Indian companies in Central-Eastern Europe because (mostly) German companies are a lot more comfortable if they can meet the dev manager on any given day and fly home for supper.