My main question is whether this is still helpful in an SGD context?

I.e. if you have $f = \sum_i^n f_i$ and with every batch you make $f_S = \sum_{j\in S} f_j$ for that batch ... being able to take a step which definitely causes $f_S$ to decrease doesn't necessarily cause $f$ to decrease / the window for $f_S$ may be different than the window for $f$?

If it only really makes sense for cases where you can evaluate the whole function of interest at every step, which tend to be lower-dimensional anyways, then I think an interesting comparison would be to quasi-newton methods like LBGFS. I.e. if you're involving the taylor-series machinery, when should you use it to pick the direction to step (normal QN) vs when should you use it to pick only the step size (this work)?