Just two things to add:

1) Often the "performance critical parts" are actually 60-80% of your code. This is the problem with the "don't optimize prematurely" philosophy. Think of it in advance instead. Do you need the highest possible performance? Are you sure? How fast should it be then? Per request? At setup? Per data point calculated? Choose your tools accordingly.

2) For numerical code I'd actually choose Modern Fortran over C. Much less to do until it's fast.