I've seen estimates of 40-60% of all households not paying anything in taxes even if they are filing them. Conditioning voting rights on having paid taxes would disenfranchise a huge swath of low income people as well as people with accounting strategies to minimize their taxes. It'd certainly wouldn't help the US's underwhelming voter turnout.
I agree with your overall point about rights without duties.