This is a great answer, but it's also very important to remember that the IP range you're locking down to may not be the CIDR block you think it is if there are BGP or rDNS shenanigans occurring.
Sure, IP address is just one more obstacle, but one that is entirely surmountable for a determined attacker.
I think that in my threat models "mistaken BGP entry" is not something that happens because of determined attackers, it's something that happens all the time because people is people.