Since this is the internet, I am free to reply with hearsay... I read that if you fall asleep that fast, you might not be getting enough sleep the previous night.
OTOH, ~90% of the population uses caffeine regularly, and given that it takes ~48 hours to completely purge your body of caffeine, I think that saying is only relevant to caffeine drinkers.