You can further be sure of this by pressing the On button while the power supply is disconnected. Ofc make sure it's always off when you connect or disconnect the power supply.
Depends on how the on button is implemented, and the power management of the system.
On older devices I would expect this to be more reliable.
Indeed, this used to be my "secret trick" for laptops that wouldn't power on: Disconnect the battery and power supply, hold the power button for 30 seconds, then power it back up. Worked every time.