If Google didn't penalize sites that don't use AMP, and instead just measured response, paint, etc times and such, I would agree with you.