http://www.blackhat.com/html/bh-usa-05/bh-usa-05-speakers.ht...
As far as actual implementation, I'm not aware of a system that does this. Although honestly some researcher has probably implemented such a system in a paper somewhere. There's probably a commercial network IDS that does it too. I haven't been doing this long enough to be an expert in the field, so I can't confidently say what the true state of the art is in behavioral detection. I'm pretty sure there are no open source projects though if that's what you're asking.
If you're thinking about implementing your own, I'd say start at the network level and write a simple program that sets an alarm off when a specific host starts responding to HTTP requests. If you want to get complex, use Wireshark to log all your network traffic for a month. You can then use Bayesian learning to determine whether new traffic is out of the ordinary. SSH traffic is problematic, so if you need to allow such traffic you'll have some additional challenges to overcome.
Starting an open source project related to this is on my TODO list, but I'm focusing on demonstrating the hopelessness of signature based detection first.