The traffic lights are optimised for throughput during peak times and optimised for travel time during non-peak times.
Throughput optimisation has the lights cycling at a constant rate, 60 seconds in one direction, 60 seconds in the next direction and so on. It's easy to tack the pedestrian cycle on to that (pedestrians walk in parallel with the traffic, though I've also seen lights which give the pedestrians their own cycle, and pedestrians can also walk diagonally across the intersection). The cycle might also be synchronized with other lights.
The Travel time optimisation for off-peak is completely different. Lights are always stuck onto a single direction (the direction with the most traffic). When a car triggers the sensor in the perpendicular direction, the traffic light quickly cycles and gives them a green light within 10 seconds. The green light only lasts about 5 seconds, barely enough time for a second car to get through the light. It's impossible to get a third though. But in typical night time traffic, this mode makes the traffic light only slightly slower than a stop sign.
Even if a pedestrian did arrive at the traffic light at the same time as a car, there is nowhere near enough time for the pedestrian to get across the road. You can't even run across the road in the allocated time.
The button is needed for two reasons. 1) to signal that a pedestrian is waiting. 2) to request the longer cycle that has enough time for a pedestrian to walk across the road.