Because, as one of the original GUIs, when they conceived of how the Macintosh worked, you used the command keys to perform actions on files, and you could rename the file just by typing. Cmd-O was the shortcut for opening files, which makes sense for obvious reasons.

When they released the Macintosh to the world, Andy Hertzfeld realized that people were renaming items in the Finder by accident; so he added the "return" key (already used to confirm actions) as an "arming" step[1].

Now, one might ask: wouldn't that interrupt people navigating via the keyboard? No, because the original Macintosh didn't have arrow keys on the keyboard. This was an intentional choice—or, perhaps, mandate from Jobs—as a way to get users to use the mouse as their primary input device, and to force developers to create mouse-driven interfaces. With no arrow keys, no typeahead search implemented, and a clean slate for designing an interface, I'm guessing using Return to open a file wasn't a strong consideration at the time.

Why wasn't it changed? Because Mac OS X was made for Mac users, and we all have muscle memory dating back decades :P

[1]: https://www.folklore.org/A_Floppy_named_lsadkfjalhkjh.html