This is also half the story in some ways. SRV records didn't exist because the DNS didn't exist. Email "was" port 25 because that was the assigned port. It was uplifted into the WKS (well known services) model as IANA port allocations, and then rei-ified by "ask IANA for a port" and low number assignment implicitly meant a service daemon which had to setuid() to the user for specific delivery events (most things were store-and-forward)

Once SRV records were invented, at the cost of 2+ DNS RTT you can ask "hey, where do I talk SMTP to this place" and be told what port to bind to. But lots of stuff was godfathered in, great-grandfathered in. Too many people think SSH is on 22 and don't want to incur the DNS costs.

before this, its 25 for mail. 110 for POP, 123 for NTP for pretty good reasons: we didn't have anything else, except inter-host agreement which ports bound to which service, and the services were almost exclusively launched by root from init process. The trend to dropping effective UID came pretty damn quickly after people realized SMTP as root meant remote actors sending Subject: "|bad stuff; even worse" subject lines to exec() calls which did implicit shell uplift.

(I am eliding this a bit)

I run SSH on a shifted port. If I could convince myself that URLs with :port embedded were "cool" I'd move off 443 and 80.