No. Sadly, Tor doesn't work like a VPN, presenting itself as a network interface responsible for a virtual subnet you can restrict your connections to. Instead, Tor connections are plain-old IP connections from random public IPs. Which means that your hidden service needs to accept connections from random public IPs--and, therefore, to be public itself. A hidden service might not have a published IP address, but it must have one that you could, potentially, connect to over the plain internet, in order for the proximate Tor node to it (what would be called an "exit node" if it were a plain site) to be able to talk to it.

In theory, you could hack on the Tor client so that clients and servers did an SIP negotiation prior to connecting on the desired port. You'd then run something on the host which would act like a port-knocking daemon, temporarily allowing new connections on a port only in response to a request from the Tor client, and only to the SIP peer in that message.

(Or, Tor could just present itself like a network interface, giving each N-proxied-peer a virtual IP that changes whenever it regenerates its identity. "Hidden service" connections would be regular IP-to-IP communication. For "public" connections, exit nodes would need to be running SOCKS proxies, and then there could be an anycast IP address that picked a proxied-exit-node at random. Then you'd just set that as your plain-old SOCKS proxy in your browser.)