Caffeine's half life is around 5 hours[1], I'd say it's fine if you drink it in the morning.
I'm quite sensitive to caffeine and I struggle to sleep if I drink coffee any time after noon. It's a pity because I love the taste of coffee and Oolong tea :(