Madison wasn't an Anti-Federalist; he actually wrote some of the Federalist papers himself (along with much of the constitution and Bill of Rights). He did later become a Democratic-Republican, but by then 'Federalist' meant the political party, not the pro-constitution movement.