It might be hard to grok the need for this if you’ve never been in a tight spot while sailing or operating a watercraft with a crew - but having this distinction has probably saved thousands of bad incidents from happening.
This is why in the days of processed x-ray films, TV shows would more commonly hang a chest x-ray incorrectly[1], with the (patient's) "left" label on the left instead of the right. It's less common now that the example imaging is digital.
[1] https://screenrant.com/scrubs-xray-gag-joke-deeper-meaning/
The ferry is actually a very odd boat in this, btw. As I am not clear that starboard/port are fixed. The boat literally goes in both directions, such that I'm assuming I don't have to know which is the "front" of it irrespective of the direction it is currently traveling.
I get your point, that things can be difficult. And I appreciate having jargon that is specific for an attempt to make things clearer. I just find this as the sole explanation kind of unsatisfactory. Seeing the history of the terms makes a lot more sense.
Edit: I found more discussion on this by searching for "double ended" boats. Port/starboard do indeed change depending on the direction of travel from all results I've found.
To be fair, they may not be 100% symmetrical. I am not measuring. They are close enough that I can't imagine it would be a good use of the terms. (That is, when the VAST majority of the time the bow is the front of the boat, having a few bigger boats where I have to inspect it to know which is bow and stern would be a bad idea.)
The car decks and the sundeck appear to be mostly symmetric other than labelling, and maybe elevator placement? The machinery under the car decks is likely somewhat assymetric too, but I've never gotten to go down below.
From what I understand, on ferries (and similar "two-sided" working boats), port and starboard get switched around when the boat changes direction of travel, usually when docked (and its navigation lights also turn from green to red and vice-versa).
I think it's just historical.
You hear a cry “man overboard on the left!”
Where is the overboard person? Are they to your left, the ships left, or the left of the person shouting?
If the cry is instead “man overboard on the port side!” there is no question.
This is about as worrisome as "you are the fire department, and you are told you need to rescue some people on the 14th floor. Quick, does that mean you go up 13 flights of stairs, or 11?" Realistically, you aren't getting it right on spur of the moment decision making.
I think starboard and port make more sense here because they only describe parts of the ship, so the meaning of the terms is completely unambiguous.
A crew (any kind of crew) requires exceedingly clear communication. "Ship left" and "ship right" could work, but what if the environment is noisy and you're not sure if the first word was "ship" or something else? Domain-specific, unique terms are fantastic tools for crews.
Speaking as somebody who sails occasionally, to me "left" and "port" are just very different concepts. It's sort of like driver's side and passenger's side of the car, but more intense. So I think the desire to use "left" mainly comes from people who don't spend much time on ships, an audience, not particularly important to sailors. Sort of like how non-programmers might struggle with some of our technical terms, but we'd never change.