Don't worry, plenty of us would consider that rude :) It's really the same as checking your phone during a conversation. Is it useful for me to give you my full attention if yours is constantly changing channels?