The vast majority of music has what's known as a meter, which is a cyclical pattern. Just like we can think of time in terms of a neverending stream of occurrences, or choose to think in terms of a 7-day week, we simplify the understanding of musical events by counting in a regularly repeating meter.

This song's meter is in 4/4, which means you count 1-2-3-4-1-2-3-4, etc. along with the very pulse you feel. Tap your foot. Count "1" when he says the word "Come [with me]". You'll find that they clap every time you say 1 and 3.

What he does is the equivalent of cutting out, say, Sunday from one week of the calendar, and so their regularly repeating clap now falls on a different day than it was falling on before. Does that help?