- There are special cases where it is feasible to do a calculation 'by hand' (actually with a calculator or a much simpler program). The complicated program should get (nearly) the same answer as the hand calculation in those special cases.
- Any solution should have certain properties which it is feasible to check 'by hand'. For example in physics there are often conserved quantities (energy etc.). Or the output should change as a known function of one of the inputs, if you have a series of test cases where only that one input varies.
- Instrument the program. Write assertions that check intermediate results. Each intermediate result should be a known function of preceding intermediate results - write code which checks that.
That last option is similar to the formal verification recommended in another answer. In flow-blown formal verification you try to write a proof that the intermediate result is the intended function of the preceding intermediate results, for all allowed inputs -- and then you produce a chain of such proofs from the input to the output. It is often much easier to just write code to check the intermediate results.