They have to give back the principal, but what about the interest earned from their bank?
Banking errors are clear cut. It was never yours in the first place.
Somewhat frighteningly, all money you have in the bank belongs entirely to the bank and not to you. The most they owe you is a promise to pay, but by making a deposit you give them ownership of the money.
So, if money you deposit isn't yours, certainly money you didn't deposit isn't yours.