The OpenBSD developers are not too thrilled to hear about these sorts of issues, but looks like sysupgrade installed sets I didn't have before (x*66.tgz, game66.tgz).
As noted elsewhere, the developers recommend installing all sets. I personally prefer to avoid installing the X sets on servers.
And yeah, you're right, even light criticism on the OpenBSD lists is met with a very strong and negative reaction. That said, I think improvement suggestions that come with a diff and a constructive attitude are generally well received.
Also sysupgrade is a pretty brief ksh script, so fairly easy to see what it's doing.
syspatch(8) also assumed them to be installed IIRC.
I hope they're real complaints and not i.e. we need install menu changed from "install xserver [x]" to "how u liek the xserver? now[ ] never[ ] now with extra firefox, please[x]"