Most of the times one would learn a model by just changing a single feature and then doing the whole sum made no sense.
A good example is learning a sequence of decisions, where each decision might have a cost associated to it, you can then say that the current decision depends on a previous one and vary the previous one to learn to recover from errors. If previous decision was bad, then you'd still like to make the best decision for current state.).
So even if your training data does not have this error-recovery example, you can just iterate through all previous decisions and make the model learn to recover.
An optimization in that case would be to just not redo the whole sum (for computing the decision function of a linear model).