The main reason Google caches the pages (when a user comes from Google search) is so that the page can be preloaded by the browser before the user even clicks. This preloading is part of the reason that the experience is so fast.

If Google preloaded the page from a non-google server, there would be a few problems. The most important one is that it would violate the user's privacy. That server would have in it's logs a request from a user that didn't click on the article, and could intuit information about the user's searching behavior from that request.

Less importantly, without the page being served from Google's cache, there is no way to guarantee it's actually a valid AMP page. The only thing Google knows is that it was valid AMP the last time it was crawled, which could be days ago.