Camera hardware is easy...throw it in, done. It's the software that is hard. People expect more now. Bokeh, auto HDR, stabilization, scene recognition, smoothing, night mode, and much more. If you put best of class lens on a phone, and it just shoots RAW, everyone is going to hate the camera.

As for the other parts of the phone, well, they just aren't as demanding, OR people just don't care about it as much. People will pay a lot of money for a camera that is outstanding, vs say, a phone with the clearest voice imagineable.