1. You don't really have to work for anybody so you can sort of do whatever you want. The costs associated with starting a software related business are almost zero. That is basically not true in any other field.
2. Programming isn't an end in itself unless you want it to be. People learn to wood work in order to make things, just like people learn to program in order to make things. I have mostly never learned anything CS related unless I specifically needed to learn about the relevant topic in order to make something, or check the security of it, or try and learn a faster way to make it. With the exception of some of the higher math stuff this method will teach you nearly everything you really need to know. It sounds like you have already taken up through linear algebra so the math part is covered.
So if you hate making things than I guess CS isn't for you. But right now I would say programming skills allow you to make more things in a wider variety of areas than any other skill set.
No one is forcing you to make a stupid app. You could be doing microcontroller programming in a subset of C or even assembly if you want to be involved in making almost every modern device. You could try and make something like google glass, or the oculus, or a lifesaving medical device. You could involve yourself in the world of 3d printing. You could do molecular simulations to push science forward. You could make videogames. You could work with bitcoins. You could probably work on rocketships if you hit gradschool for maybe 2 years.
You can start a business if you want and do more of the client facing stuff.
With programming the possibilities are really open right now.