I might halfway pause and start a bad reference with, "Well..." and let the manager guess if he's perceptive enough, but I'd never speak ill of a former staff member. Too much chance of it biting the company back - it'd be pretty damn irresponsible of me to do that. But not calling back? Yeah, I'd do that. For great people? Callback the same day I get the message, no matter how busy, at least for a 90 second chat. Maybe I'm not everybody, but I think most halfway competent managers go out of their way to take care of the great people they've worked with.
I've got fierce loyalty pretty high in my ethics, but there's a lot of pragmatic reasons too. On any one event, you could blow it off, but it's pretty obvious on the whole who goes to bat for their people and who doesn't. It's not something you make a calculation on - "Hmm, no one will know or care if I don't help this particular time." It's a way of life - take care of people that take care of you. Feels good inside, world sees it, recognizes it, and treats you well. A good thing, top to bottom.