As both a traditional sports fan and esports fan I agree that it can invoke the same responses.
I see the "lack of physical skill" argument a lot and my typical argument against it is endurance and mental performance. Esports players at their peak train for 8+ hours a day and can compete for 4+ in some longer events. Doing anything with your body for that long takes physical and mental conditioning.
I also see a lot of traditional sports detractors saying "What's the point of just watching someone play the video game, seems boring" and it really blows my mind how much of a disconnect there is.