This is the kicker, thanks. I'm using a (non-Apple) mouse and changing the settings didn't seem to have any effect.
You should still be able to make it disappear if you change it to "When scrolling".
I tried that setting before and it doesn't seem to do a thing when you have both types of devices connected.
I keep a touchpad on the left and a mouse on the right (had the extra hardware then got used to the config), plus the one on the laptop itself, and they're hidden with the "Auto" setting.