Usually a crossing will instantly switch when the pedestrian button is pressed, if enough time has passed since the last "walk" cycle.
Having a stage where walk is enabled when there's no pedestrians around wouldn't much help pedestrians, and would introduce inefficiency in throughput. And obviously, drivers can't press a button, so it makes more sense for controls to be accessible to the pedestrians.