The main breakdown products of aspartame are aspartic acid and phenylalanine. You may have been born with a genetic mutation that makes it possible for you to taste phenylalanine, which tastes bad.
It also gives me headaches, to the point the rare occasions I drink soda, I rather have HFCS than aspartame.
It seems impressive that aspartame would be broken down quickly enough in the mouth to reliably produce a bad taste.