I think at 20 I would have been a better test taker if we both started a college class right now but that is about it. I am way "smarter" or "more intelligent" though at 54 vs that 20 year old self. It isn't even close.
IMO much is because we are blind to ET Jaynes Mind Projection Fallacy in this area. We think of "intelligence" as a physical property like mass that gets weaker with age like muscles or bones. It is just such a given in our thinking and language that we can't help it.