That way, sites like HN are boosted to the top, while sites which take 10 seconds for their JS frameworks and ads to load before showing text on the screen are penalized.
They get to have their cake and eat it too, and there was no incentive (before AMP) to reign that in.
Google appears to have decided not to, in an apparent effort to boost their ecosystem, at the cost of the open web.
It would seem like the reality is that page speed is not an important factor, and that re-tuning that factor would have been a better solution if Google's primary concern was for the health of the open web; not just Google's ecosystem.
I touched on this confusion elsewhere in this thread, but AMP is actually not a ranking factor. The carousel is a vertical which has been integrated into the page.
However, I do agree with you. It's confusing to end-users when it's described as a non-ranking factor because it still appears on the search page. It just doesn't influence organic search.
I would also like to see page speed become even more important, as performance is very important to user satisfaction. I spend a lot of time trying to minimize bits wherever possible.
It's not just pure loading times either, but perceived speed too. Small effects like animations can make a page feel faster, even if it's technically slower.
Personally, I wouldn't be surprised to see the carousel disappear once Google deems that AMP has become self-sustaining. I think they're using it as motivation to get publishers on board, and then they'll fall back to regular organic ranking afterwards (to which AMP pages will still benefit).
But in the mean time it's causing a lot of consternation.