If the driver links to the Linux kernel underneath, porting shouldn't pose a big problem.
Oh... Isn't distributing proprietary drivers linked to the Linux kernel a violation of the GPL?
Problem isn't so much kernel level drivers, but user space. Or undocumented proprietary APIs to firmware. I suspect it is not a problem for most components. Most phones are just recombinations of the same components. The problem areas would be the radios and possibly the graphics.
Check the tear downs of the HTC Sensation and the HP TouchPad .. virtually the same device inside. ;)