I mean if you move down 3 notes from any Major key you have your Minor key with the same exact notes, like C Major and A Minor.

I usually compose with a Major or Minor key in mind, but it's kind of a reference and not a be all end all. I like sharp 4's which work great usually and don't fit in either key. A flat 7 sounds great in a major key too, usually. I agree with you mostly, and i am annoyed by people who are adamant about the rules of a key, but it helps me to have a key in mind to both play and compose, even if i break the rules of it.

I find improvising on piano / guitar is nearly impossible if you don't know the key, even after many years of playing both.