That delay should always be there, even when the window regains focus after being out of focus. You just usually don't notice it, because it's usually already done until you have your mouse on the button.