Exactly the same experience here.
I have ADHD. Caffeine seems to give me a parallel (albeit lesser overall) experience to taking my prescribed Adderall.
Most people around me have diagnosed ADHD, makes me wonder if the reason why I'm in such a group is because I have ADHD as well, but I don't really care for medication as I never had issues handling it on my own so not planning on getting diagnosed.
And if I do, I actually prefer it since it allows me to go on for days without a break discovering solutions to very hard problems.
How much caffeine do you consume regularly? Sounds to me like your body potentially has too much already, and is reacting by making you drowsy. Note that caffeine is also in non-coffee/tea related beverages like sodas (e.g. Coke/Pepsi/...), or sometimes certain pain killers (e.g. migraine pain killers frequently contain something like 50mg of caffeine).
I mostly drink decaf. Not a soda drinker. Tea with sugar mostly or just plain water.