However, this doesn't mean that an HPC application has no need to verify its input, fail gracefully if something goes wrong, or have to be absolutely rock solid because it's running for days (or weeks in some cases) while being dead-on the results of the problem evaluated.
You're right. Correct is the king in HPC, but it's a king which doesn't exile speed. What we do is to solve a couple of known cases correctly in MATLAB or something similar, then try to hit the same results first (or be better if we can do that), then make it iteratively faster without deviating it from a bit (Same result with 1e-32 precision, IOW).
This "comparing with a known truth" process is a great indicator of calculation sanity. Other parts are iteratively tested with Valgrind and custom test suites, from function level to end to end, automatically after each build. End to end tests are very expensive in the Valgrind memory profiler (esp. if you're testing multi-thread code), so we do these weekly generally, but I think the idea is clear.