I fixed it by wiring a simple toggle switch into the the power line for the stereo and mounting it in the dash. Problem fixed. "Off" shuts off the stereo instantly.
You might be able to read up on your model/package and do this, or consult a car stereo installation shop to do this.
The Honda CR-V is like you say, it will only turn on if you had the radio on when the car was shut off (makes sense), but turns on long before it is capable of responding to the off button.
The Chevy Bolt infuriatingly turns on (almost) every time you get in the car regardless of whether the radio was on or not when you last used it. For the life of me I cannot figure out what heuristics cause it to (very rarely) not turn on. At least its off button is functional at the time it starts playing.