The fact that the OP isn't hosted in EC2 but in "locations [that] map well to AWS regions" throws the whole thing off. It doesn't matter if their servers are across the street -- Amazon likely doesn't have good latency and routing information for datacenters it does not own.
R53 exists to make DNS-based global server load balancing optimal for EC2 servers. They have little incentive (and limited ability) to make it right for random datacenters. And OP's intuition is right -- balancing at the DNS level means balancing at the DNS resolver level (leaving aside edns stuff), which adds another layer of coarseness out of Amazon's control.
Either way, data wins arguments and I'm prepared to be wrong. To test this more thoroughly, OP could spin up an instance in each EC2 region their real systems "map well" to, proxy to the real systems, and see if the routing gets better.