Even if you rarely use the brakes, seems to me like a good idea to keep them functional. And based on my experience working on old, worn out shit, the less something is used, the more likely it is to get corroded or gunked up and not work.
My Nissan Leaf's brake rotors rust up and I have to do a few hard brakes every month or so to clean them up.