Lets say it's 2011 and you have $50,000 saved for a down payment on your dream house. I steal that money from you, but return you ~$56,000 7 years later (I've adjusted for inflation).
Technically you haven't lost any money.
But you missed the opportunity to purchase your dream house during a dip in the housing market while interest rates were low. Now, not only is your dream home 30% more expensive, interest rates are higher. You also missed out on 7 years worth of mortgage and real estate tax breaks. Your credit score isn't as robust either, since you don't have 7 years worth of mortgage history on it.
You are way behind despite getting every penny back.