Depending on specific 'simple' cases this can be messy but effective.
The same deal happens for decision trees. (Axis parallel) Trees have a hard time fitting linear functions, requiring many modes. One can use essentially the same solution.
BTW very informative commentary. The things that you bring up really needed to be brought out.
Let me give a simple example. Say one wants to reflect 2D points along a line at 42 degrees. It's eminently doable in canonical basis but it's quite messy. Instead one can work off a different (rotated) basis such that that 42 degree line is now the x-axis. Now reflection is trivial. However the post reflection coordinates are in the new basis, so you invert the basis transformation to get your answer in the canonical basis.
For solving linear ODE with fixed coefficients you have a similiar situation where you transform to Laplace transform basis. In this basis the coefficient Matrix is a diagonal matrix, so it's trivial to solve. Then transform back (not always easy).
For de-noising tasks you need a basis where noise and signal have drastically different coefficients when expressed in such basis. Then it's easy to knock out the offending parts.
Harmonic functions are smooth and have the 'averaging' property. If you know beforehand that your desired unknown function is smooth in a similar way, Harmonic functions (for the corresponding smoothness measuring linear operator - the Laplacian operator) is indeed a very good choice.
As a counterexample if you want to denoise a photograph but retain edges, then Fourier is not very convenient because both edges as well as noise have 'high frequency' heavy representation. If you throw out high frequency components, the edge pixels are going to be collateral damage. The problem is that in this basis, edges and noise look the same. Shearlets, curvelets will do better here.