Sounds like you are looking for a self hosted ZeroTier or Tailscale.
[Edit] Answering my own question. Custom SAML providers are only supported with the Enterprise edition. [1]
FWIW, tailscale is technically not free once more than 1 user, its a commercial service at the end of the day not some self-hosted open source application. They have extremely generous fair use policy in my experience and won't bill for a small number of users though. This means custom SSO is likely only a real issue for paying customers to begin with, although I can understand the frustration for the few users who do want it on the free tier.