In my experience, it's pretty common for people to bring up the fact that something is legal to avoid the question of whether it's right.
So some people bring up the founding principles of our nation and the case law built around it because it's the best kind of common ground - agreed upon societal basis/"social contract" - they can think of to explain why some immoral things should be illegal but some immoral things shouldn't.