Your first point is fully correct. The 'expert system rules' didn't work very well.
On your second point, "what does work", actually doesn't work very well either. You can use such things okay for first-cut, say, what J. Tukey called 'exploratory data analysis', then, to be followed by 'confirmatory data analysis'. The first part is okay, but the solid stuff is the second.
Then there's more: See what assumptions hold in the practical problem. Use these assumptions as the hypotheses of theorems with proofs. Use the conclusions of the theorems as results you now know about your real problem.
In real problems with a significant role for uncertainty, look for cases of probabilistic independence. For this, learn the high-end version of what a random variable is from, say, Cinlar, Breiman, Shreve, or Diaconis. Then for random variables X and Y, if knowledge of X doesn't help predicting the value of Y, then X and Y are independent, and can often come to this conclusion just intuitively.
Cases of independence can partition your problem into 'independent' pieces.
For more, look for some good examples of how to apply math in physics, chemistry, electrical engineering, and operations research. For statistics, sure, look at what agriculture did with the general linear model or, if you will, analysis of variance.
You will have little hope of making progress in using applied math just within CS or computing. For the progress, have to look to other fields that have made good progress with applied math. Eventually it will be crucial to have a good background in math, say, Halmos 'Finite Dimensional Vector Spaces', Rudin, 'Principals', measure theory and functional analysis, and then probability, stochastic processes, and statistics based on these.