I'm also a remote worker, but I work (mostly) on my local business hours. I'm in Brussels and my team is in California and I'm usually working a bit later (until around 7-8pm) to overlap with them and there's the occasional late night meeting, but rarely. This is a compromise on both sides (I'm working later than normal and they try to schedule the meetings early morning) and I think it's the only long term workable solution when you have teams in very different timezones. Otherwise you'll just burn out and get tired of having no social life and probably you'll health will be affected too.
Besides, do you really need to talk to them all day long? If you're a developer, can't you just have some "sync-up" meetings every other day or so, during their morning?