That's my concern with fully remote work. You're not at the office, so only the people who work with you directly even know your name, and you don't form a personal connection with anyone. Even the people you work with directly have only seen you a handful of times.
When the time comes to cut someone loose, you're the obvious candidate unless your work is head-and-shoulders above the other people. It's a lot harder to fire people when you've been swapping stories about the kids and how about that game and whatever over the watercooler for any length of time.