The worst: pressing Enter to rename a file in finder. Command + Down arrow to open it. WTF!
Whether renaming as “enter” is a good shortcut or not, I think the navigation ones are perfectly reasonable in that if you learn one of them, it’s opposite is easily guessed.
After using both for a long time (Windows for 30 years (!) and Mac OS for 16), I actually like the Mac OS way a bit better. In my usage renaming is a very common operation and I don't have F keys on my usual keyboard (renaming in Windows is F2).
But what I mean is that Finder navigating uses an unintuitive key chord you have to learn, while renaming uses a single key, and a large and very common/obvious/intuitive one. This does in fact mean the UX is declaring which one they expect to be more common. I'm not saying renaming is uncommon, but for me it's nowhere near the usage of navigating into a folder. Are you saying you're renaming folders more often than browsing into them? (How do you get to the folder you want to rename?)
Both Windows and Linux GUI folder browsers use the Enter key to navigate, and so do many other tree browser UIs including our web browsers. It's very common and very reasonable to expect Enter to navigate to the thing you've selected. This expectation and ecosystem of navigating this way means that Mac's Finder stands out as different and unintuitive, even within a Mac-only set of applications. I love my Mac, and I've also used both Windows and Mac since they first existed ... both for 35 years. But today's Finder is definitely weird and anachronistic compared nearly all other modern file browsers.
Because you're always in an app in macOS, Cmd+O is the more consistent choice: every app you're in will use Cmd+O for "open." In Finder, what you're opening is apps or files, so it works.
Return for rename makes sense in a more subtle way.
Besides opening a file, there are two "main" actions you'll use in Finder: renaming files, and quick-playing files. In a dialogue box, you'll either use Return or Spacebar (the former for triggering the highlighted button, the latter for another button you highlight with the Tab key).
Consider again the consistency-across-apps argument. "Rename" isn't a super common command in all kinds of apps (a DAW will allow you to rename tracks, but that idea doesn't really apply to a text editor or IDE), so using Cmd+R in Finder for "rename" isn't necessarily great for consistency, considering that apps in general will almost surely bind something to Cmd+R ("run" or something like that).
Due to the aforementioned dialogue box pattern, Spacebar or Return are the obvious choices for rename, simply by virtue of being used so much.
Spacebar technically makes more sense because quick-play is adjacent to opening a file, but the ergonomics of "Spacebar = play" across so much media makes it the obvious choice for that. So, Return it is.
I’ve used macOS for 16 years, hitting enter/return still feels weird.
Return is a holdover from typewriters. It means "return the carriage to the left margin". I'm surprised the Mac keyboard doesn't say "Enter". It seems like a much more accurate word to describe what the button does.
Yes, "Return" applies in the context of editing text (return the cursor to left margin) but not in any other context on a computer I can think of.
I wonder why they continue to label it return and not enter.
/end random musing
edit: Downthread, someone provided a link to a StackOverflow page explaining how MacOS treats enter and return differently in some rare instances (it's program specific).