No, it isn't. It's illegal to sell it across state lines, or in some places to the general public. But in most states, you can buy it at a farm, or other licensed facility. It isn't like heroin, or even pot, and you won't get in trouble for possession.
http://www.farmtoconsumer.org/raw_milk_map.htm