So you didn’t own everything. Google owned the IPs and network.
It’s the same colocating in that your network providers can be shut down.
Even if you do there's still more to it e.g. blacklists of different kinds e.g. known proxies, spam listings, etc that would still mean you need to kick out your client.
Then there's the bad clients you need to deal with e.g. if they send DoS from your network. Even if you don't get a report it will eat up your limited resources and you may be subject to revenges and bring down your infrastructure.
Conclusion: there's lots to it but so does every business. That's just life.