You can enjoy Pac Heights, Noe, the Dogpatch, the Sunset, the Richmond, North Beach etc. etc. just fine without ever making contact with downtown, the same way most people don't hang out on 5th and San Pedro in LA and can still enjoy Silver Lake. You can pretend it doesn't exist and never cross paths, unless you really insist on spending quality time on 6th and Mission and other fine intersections.
Would I like it better if the Tenderloin was more Fifth Avenue and less World War Z? Sure, but that's not happening with the way SF politics work, so I might as well live with the cards we were dealt.