My Casio CMD-40 smartwatch has a calculator and programmable TV remote. The CR2032 coin cell inside is enough for it to last about 2 years. The display is always-on, and I use the calculator often, although admittedly I rarely use the IR transmitter. So apparently sensors use a lot more power than an LCD and calculator.
Even using your IR transmitter to send a short burst every 5 minutes wouldn't be as much power as if you had an IR receiver that had to listen to incoming signals. But still, start using the IR transmitter every 5 minutes and you'll see a dent in battery life
Also, the watch has a clock frequency measured in kilohertz, and the processor is probably some dumb as rocks 8051 variant or something with a tiny amount of static RAM. Big difference in power consumption compared to a 32-bit ARM core.
Cortex-M0s are pretty competitive with that power regime though these days, especially if your application (like the watch) will allow you to run the M0 at sub-1MHz speeds.