Something like mupdf (which has already been ported) combined with some button/touch bindings and perhaps a touch-friendly zoom feature seems to me like it would do the job perfectly... and be an easy weekend hack for much of this site's target audience?*
I think you'd have to look long and hard for an e-ink reader with truly decent out-of-box PDF support and no serious failings in other areas — my search so far has been unfruitful, but please do tell if you've come across anything which fits the bill.
*edit: If someone actually does/has done this/something similar, please post about it here!