Huh, I've tested on a bunch of devices/connections and haven't encountered that. Do you know what causes AMP to be that slow for you? I'll take a look at serving non-AMP pages by default. It will require tweaking how image inclusion works.
Here is information about some of the other problems with AMP:
https://www.theregister.co.uk/2017/05/19/open_source_insider...
https://danielmiessler.com/blog/google-amp-not-good-thing/
https://ethanmarcotte.com/wrote/ampersand/