You are very optimistic about publishers actually making the websites fast. (like just how they did before AMP existed).
Disc: Googler but not associated with AMP whatsoever.
Publishers have shown that they can devote developer effort when there is a clear ROI. Google just need to make the criteria for inclusion in the carousel understandable and measurable.