Keep in mind that the word "dumb" gets thrown around in all sorts of situations and has no objective, easily-verifiable test for "dumbness". It's entirely plausible that merely "making a decision I disagree with" could be construed as "dumb". Imagine if, say, Biden and Fauci had decided to disenfranchise anyone who refused the COVID vaccine as "dumb", or, worse, Trump and RFK decided that taking those vaccines made you "dumb". See how easily this can be abused?
Even with that aside, the natural consequence of disenfranchising "dumb" people is that you want to fuck them over somehow. No thanks.
No matter what kind of voter you particularly would like to target, the people administering the poll tests will (ab)use them to disenfranchise whomever they please.
Even if you believe the people in power who administer your elections would do this the "right" way, if you give them power, then the next people who come along -- who you may not agree with -- will also have that power.