Best case scenario: passed over for promotions, cut out of important work discussions, generally ostracized. Worst case scenario: outright abuse.

Just to give you a taste of this: In a previous job in Japan there was a work party on the weekend. I told my boss "I can't got, I have something to do" (A very normal way to refuse something in Japan. The expectation is that no one will ask what your vague reason is). This boss tries to pin me down on exactly why I won't be spending my day off on this event. This leads to 6 months of going over my work with a fine tooth comb, review periods and so on.