The right way to understand the behaviour of computations on a real machine is inherently mathematical. This is why, for example, we have an entire discipline of numerical analysis, just for starters.
Programmers very commonly make elementary mistakes in computation precisely because they don't understand this, and don't approach the problems mathematically. Ignoring overflow, or pretending floats have infinite precision is antithetical to thinking about these problems mathematically.
Conversely, any mathematician worth their salt will approach computation on a computer not as a black box, but as a concrete implementation. Mathematicians make lots of programming errors (some of them quite egregious and easily avoidable). Misunderstanding floating point representations or the overflow behaviour don't tend to be among them in my experience.
By the way: Knuth used a hypothetical machine to avoid the quirks of a concrete one distracting from the discussion.
I am not completely discounting Numerical Analysis here. Mathematicians can make entire mathematical models of a machine but it will be at a certain level of abstraction. In practice there will be abstraction leaks due to some quirks of the implementation or real world dependencies.
So one should not merely depend on the mathematical model.
My confusion at your (and others, not meaning to pick on this one) response is more fundamental than that. There is no way in which thinking about these things is somehow divorced from thinking about things mathematically.
To me "Merely depending on a mathematical model" isn't a problem that actually happens much. Failing to think about these things, on the other hand, very much is.
Well, profiling IS also math.
If anything, it would be "optimizing" which is programming proper.