You're going to tend to have unevenness there because you're returning results to recursive DNS servers, not clients; so maybe you get unlucky and return A to more large ISPs; their recursive servers serve more clients per lookup and you've got lumpy results.
Then, if you return both IPs, you're very likely to eventually come across some people who sort multiple returned A/AAAA records and will prefer one that's "closer" to their current IP. Because if you're in 1.0.0.0/8, it's certainly going to be better to connect to 1.2.3.4 instead of 5.6.7.8. Same thing happens in v6 land, of course.