Agreed. The latency on the android devices I've tried prevents them from feeling real.
However, there are a few Android phones that are particularly laggy. The Galaxy III comes to mind - when I last used one. It didn't feel anywhere near as responsive as my iPhone.
I have the Verizon SGS3 (albeit running CyanogenMod) and the responsiveness is very good. TouchWiz (the Samsung Android overlay) probably adds some latency and CyanogenMod uses a newer kernel (3.4.xx vs 3.0.xx) so that might also contribute to the difference...