Public keys should be rotated which would expire links. I think just having trust on first use would bridge most of the gap for local devices.

.. Now if we could get local socket support for talking to your own device and avoid arbitrary port reservation on localhost, that would make me happy.