The US tax code also rewards people getting married, typically when the partners have very different incomes (see marriage bonus).
Marriage penalties and bonuses are a consequence of joint filing combined with a progressive rate structure. They continue to exist because to get rid of them would require giving up either join filing or progressive rates, and it is very unlikely that people would go for giving up one or both of those.