I use another niche keyboard layout (Workman) and have never really felt the urge to rebind anything in vi(m). I can definitely see wanting to rebind the standard nav keys which causes cascading changes and can cause you to lose a lot of the mnemonic advantages of the commands.

My solution to that issue is "just" alternate bindings which obsolete both the standard and vi navigation keys. At the home row, I have the arrow keys on my right hand and a "nav cluster" of my own devising (page up/down, tab left/right, workspace left/right) on the left.

Definitely not for everyone, but a possible way to have your cake and eat it too.

bashrc:

    set -o vi

inputrc:

    set editing-mode vi

Now bash and anything using readline will use vi key bindings. This Just Works for most of the common cli tools, e.g. the python shell and psql, but there are exceptions. In the past, the mysql client hasn't respected these settings, and there's also a node library called vorpal that is reinventing readline, poorly and for no obvious reason.