Well, X11 hasn't been provided by Apple since 10.8, about 7 years ago. So it wouldn't be touched by OS upgrades anyway (it's a separate product).
/sw isn't provided by Apple either, but they provide it so that Fink will work.
On my older High Sierra box, /usr/X11 is a symlink to /opt/X11. Catalina ships with /usr/X11 symlinked to ../private/var/select/X11. So it should still be possible to get X11 to work.