I'd imagine the hard evidence connecting him comes from the raids. They clearly managed to locate the hardware the site was running on, and it's going to be a bit harder to hide connections to that.
However, the document doesn't seem to state who the VPN provider was or if he was the customer tied to the VPN account (probably avoids this info because the VPN is account is in someone else's name). So again, can't prove via this route that it was him.
I'm curious to know how they managed to find the Silk Road server though..
That is not, by any measure, careful. That's awful OPSEC, and what got him in the end. He thought Tor to be inconveniently "slow", and used VPN instead.