If I actually knew the person, a "no" with no explanation would virtually always have negative consequences. To a boss, it would likely be insubordinate. From an in subordinate, the same and would likely elicit a "I beg your pardon?" From a friend or colleague, it would cause me to question our friendship.
In the modern world, politeness is largely built on a willingness to explain and give verbal consideration. Not doing so will often (if not usually) feel disrespectful or like some sort of power-play.