Savvy individuals could manage it themselves, by creating their own servers and encrypting the traffic and only granting access to trusted individuals. Most people, though, would have a hard time navigating that. A centralized (or perhaps semi-centralized, a la Mastodon) could help a lot on that front.
Perhaps something comparable to Google+'s Circles concept. I always thought that was a great idea which was unfortunately rendered completely unworkable by the unfortunate accident that it was implemented by Google.