We should not define people in terms of what they are "not" or lump them into a group of who they are different from ("minority"). So no, if you are white you can be just that.

Maybe it doesn't have as much play outside the U.S. but the term PoC seems to work well here. For the whole semantic argument I'll point to http://en.m.wikipedia.org/wiki/Person_of_color .