The point I was making was that the proximate node to the hidden service--the last one in the onion-routing chain--connects to its destination by using its public IP to talk to the hidden service's public IP. From the perspective of the hidden service, the node proximate to it in the onion-routing chain is a regular Internet peer, which is impossible to distinguish from any other regular Internet peer.
In the end, what Tor gives you is a proxy (to a proxy, to a proxy.) And, from the server's perspective, there's no difference between a proxy and a regular client. It can't tell, by the IP, that the client it's speaking to is a proxy. And because of that, you cannot, at the server-level, block non-proxied clients from speaking to you. Because you don't know which those are.