I haven't used AMP, but my impression with all of Google CDN JavaScript files for AMP is the end user has to download them just once per version (which I agree will lead to a bloated first load experience) but on subsequent requests throughout the AMP network they'll be loaded from cache. Similar to how Google recommends you load jQuery through their CDN since millions of people already have it in cache.