Possibly; the long-lived connections eventually have to end when reloads (note: restarts and reloads are different) happen, to free up resources -- otherwise they leak. (The grace period is configurable and can even be disabled.)
I wonder whether it's technically generally feasible (possibly depends on net/http2's capabilities) to handover the connection from the old to the new server instance, comparable to what systemd does in terms of FD passing for socket-activated services.
To be clear, the new config is running at the same time as the old is still tearing down; all new connections get handled by the new config. Listeners are shared and transferred between servers.
We're all ears, though, if someone with deep knowledge of Go's internals can point us in the right direction.