just spin up a separate network namespace and macvlan device to run your ntpd in. it'll get a separate v6 address via router advertisement and ntpd will be the only thing listening on that address.

just wrap ntpd in firejail[1].

ipv6-per-networked-process pretty much brings an end to portscanning.

[1] https://firejail.wordpress.com/

Could switch over to NIST at time.nist.gov See http://tf.nist.gov/tf-cgi/servers.cgi for info/details

Granted that means trusting NIST over someone else.

The other options are getting time from a cellular modems, GPS or the most extreme running your own atomic clock.