1) High percentage posts flagged as 'reported'
2) High percentage posts flagged as 'spam'
3) High volume of posts containing URLs
4) Posts containing identified spam messages (like how an email spam filter may work)
5) Multiple accounts from similar/same IPs
Almost all spam in a subreddit I mod for is from accounts < week old. (I believe there may already be a limit on how much you can post after registration though)