Not having a math degree will impact you later if you want to get into grad school in math/financial math. If you are pretty sure you are going into software, ditch the math degree.