The fear stems from the replacement after Scalia -- if Ginsberg is replaced and Kennedy is no longer the median vote, the Court will take a sharp turn on a lot of social issues.
There is nothing preventing the Supreme Court from deciding to hear an appeal (eg, of a Clerk who doesn't want to issue marriage licenses) and then issuing an sweeping decision completely to the opposite of the new status quo.
That is the risk of relying on the Supreme Court to recognize unenumerated rights; the recognition will only last as long as the Court remains favorable to the right.