Inclined to agree here. Luke is going on about Intel ME backdoors on Twitter, but in reality there will be a far less crazy explanation for how he got owned.
Your colo provider can be thoroughly owned, your adversaries can have physical access to the server for extended periods of time and still not be able to do anything because you've denied them access to any ports that'd allow DMA.
Lots of cheap DIY options for fancy case intrusion detection going way beyond that offered by mfgs. USB camera and some tape?