There needs to be a button based passcode to view TOTP instead of just pressing one button once. That would add a layer of security. A combination of buttons and number of presses should still be somewhat added security.
Someone who steals the watch from my house doesn't have the password.
Someone who phishes the password doesn't have access to the watch.
The government agent who has exerted enough physical force or legal coercion to get me to cough up the password can demand the TOTP code at the same time.
The four number code IS the added security already.