The hash is used to find listening sockets. Source port doesn't help you there.

Destination IP is a reasonable addition to the hash function, however.