I have disabled PAM by default on all my boxes that run sshd for the last 9 years out of habit, I long ago forgot the reason why (probably because the gentoo sshd handbook entry said it was a good idea). Why UsePAM is set to yes in sshd_config by default on many distros is beyond me.