I don't own an iPad and I just bought a mini. It's my first tablet device.
Or maybe I am just completely out of touch with the US consumer and their purchasing priorities (I am from Canada)
As you said if it's only previous iPad or iPhone users there are a lot of those.
What about people with small hands ? Maybe you hand your kid a $329 device that fits his hands instead of a $500 that's too big ?
Getting and ipod touch for your kid ? Bump him up to a tablet ( if he doesn't take it to school everyday )
People that like the kindle form factor will also be tempted.