> I've come the long way around to deciding that, those people need to be included anyway.
Isn't that the point? At least from my POV, it's completely meaningless unless literally everybody is included. So yes, I'd include children, the mentally-infirm, and definitely convicts.
Think about how "universal"[0] enfranchisement came about: it has been a series of grants of influence/power, in layers, starting at the top. Barons demanded more say, and less arbitrary treatment, for themselves. They didn't get it, until they used force. Women were treated as we now treat terrorists, when they demanded a fair say.
The whole purpose of electoral systems is to deter people from trying to actually take power. So not including what you call "these people"[1] is a serious mistake.
[0] It's not universal, anywhere. Of course.
[1] The phrase "these people" is often used when othering a group of people. For example, it seems to spring easily to the lips of some Conservative politicans in the UK, when discussing immigration. It's sort of a trigger phrase for me.