This can be pretty bad in a world where AWS ELB IP addresses change regularly.
EDIT: It would be fine as long as your site only served HTTPS content and HSTS was enabled for your domain, preventing any sort of MITM attack.
They'd need to specifically gain access to the last known good IP address, which might be different depending on which DNS resolver you talk to (geodistribution, when the record was last updated, etc). I wouldn't really consider that a realistic attack vector.