Yes. Many years ago a server was gotten into by a brute force ssh script. The cracker then used a local exploit to gain root access and run all sorts of unpleasant software, downloaded my databases, etc...

I ended up just getting a new server, as I couldn't track down all the things he'd done. I started learning about server security. I reduced the number of people who had ssh accounts, and installed stronger password rules. I use iptables to block brute force SSH attempts. I use logcheck, chkrootkit, and other tools to constantly monitor my server.