See if bash's ignoredups and erasedups helps.