Those prepayment fines are basically what the bank is losing out on you. If you have a 3% mortgage and the current rate is 2%, then the bank would lose out 1% for the current mortgage period. That's the penalty.
Whatever new rate you might get is not the business of your previous loan holder.