I doubt you spend every second of your non-work time staring into space though. I like to read, for example; just because I do it on my phone in public sometimes doesn't mean I'm not finding other times for quiet contemplation.
Do you really think that's what was being suggested? I assumed they were merely suggesting to not reflexively reach for your phone any time you have a moment of downtime.
You're right though, that's likely what was meant, which I agree with. The hyperbole was unnecessary.