You can't renounce your US citizenship for quite a while after you leave. During that time you have to pay US taxes.
(Ironically, you have to pay more to renounce US citizenship, then you have to pay in all the various filing fees etc when acquiring citizenship through naturalization.)