When a store is selling 20 pairs of a shoe, they will literally look at how the person who bough the pair did their web execution. Using a something like python requests is faster sure, but most of the time looking more "real" is more important. I've written a few of these and do both depending on the company (size etc.), their website, their security etc. Some of these companies have gone as far as setting up a web game people play and the highest scores get a pair of the shoe (slamjamsocialism).

Not in every case, you're right, but it's something to keep in the toolkit. Sometimes it's easier to just check the DOM after its fully rendered than it is to explore all of the calls made by each individual site you're scraping.

What you're saying makes sense if you're looking at a handful of sites, but it doesn't scale very well compared to a universal solution for monitoring several retailers at once. It's definitely the more wasteful solution, if you are concerned about bandwidth and resource consumption, though.