Then it would depend on the nature of the taxes. If the government is smart, they would only use taxes related to motor vehicles to finance this infrastructure. Think vehicle registration, fuel taxes, sales taxes on vehicles, sales taxes on automotive parts, etc.
Whether or not it's fair enough depends on how much your government tries to make it fair.
(As a fellow cyclist I am also cognizant of the fact that roads are mostly paid for by gasoline taxes where I live, and I also enjoy subsidized infrastructure that way.)