Seems like this would be a good way to frame somebody else. Hack into a server, do some damage/steal files, and drop somebody elses public key on the server.

"But I didnt do it!" - Then why was your key on the server?