Also not OP, but the biggest hurdle I've found is communication. I feel I'm quite good at communication and I do a lot of little things to make other people feel comfortable (like making sure that my work is always visible, discussing problems and decisions I'm making on group chat, etc, etc). One thing I didn't take into account when I first started doing remote, though, is that communication is a 2 way street. No matter how good you are at it, if the person you have to talk to can't deal with you being remote, it can be really difficult.
It helps enormously if the company you work for is remote across the board because then everybody is struggling with the same issues. If you have a team where some people are remote and some people are local, the local people often don't realise problems.
I've found most of the issues arise around meetings. I work 8 time zones away from the main office so it's an extra challenge. People often plan meetings at midnight my time (even regular weekly meetings), or they will forget to invite me. Sometimes people will make decisions in the hallway and forget that I wasn't in on it. Frequently people will optimise for being interrupted in person, but not online. If I need to talk to a stake holder, I'm often in a position where they have turned off notifications on chat and are 2 weeks behind on their email. I literally have to ping someone else and ask them to walk up to the stakeholder's office and knock on their door for me.
Another serious problem is that people who don't live online often have trouble identifying with you as a person when you are remote. I've had instances where people will get really angry with me because they imagine I'm someone who I'm not. Although video conferencing helps, even 5 minutes in person can make a huge difference. Make sure that the company you are doing remote work for has budgeted for some face time every year. Also realise that this means you almost certainly have to travel a bit, unless you live close to the office.
Again with respect to having a remote job in a not-completely-remote office is that you are going to be on an uneven playing field with your colleagues. I've had a PGM once literally try to sabotage me. It's no fun and the worst part is that there is almost nothing you can do. It is insanely easy for a single person to engineer it so that you are mostly powerless. You really need to be able to trust your co-workers in a way that you don't need to when on-site. This is one of the reasons I wouldn't recommend it for someone who isn't already fairly experienced with politics -- you need to be able to keep your cool and wait for your opportunities.
The other main downside is that you are almost certainly going to be at a disadvantage for promotions, etc. Management types trust people who they can see and talk to every day. No matter how good a job you are doing, things are going to sway towards the people they connect with. When remote, it's almost vital to make sure that you keep talking with people, and maintain communication through email or chatting. You need to make sure that your name also pops up on reports and projects so that management remembers that you exist. Again, if you haven't got the experience to know how to accomplish this, then you may have a rough time.
For these kinds of reasons, I highly recommend at least starting out with either part time remote (a few days in the office and a few days remote), or working on a full remote company. I would especially avoid working from a different time zone when working with a company that is not fully remote until you have a lot of experience doing remote work.