If someone like Prince Harry just sharing his opinion that it's a waste of time is seriously damaging, that to me indicates that he may be hitting a nerve there.
Its not such a big deal from one person but if it's something that is a significant part of your life and you are constantly bombarded that it is negative then yeah its gonna have a negative effect on you.
Everyone has hobbies that are time-wasters to an extent. It isn't the end of the world if someone has the opinion that this one habit might be addictive.