You can also just lower the dose instead of following this black and white “am I a drinker” thinking. A single (1/2 double) espresso in the morning is not so much.
Not all coffees are the same. Some types of drip coffees (such as Tim Hortons) make me wonder if it's laced with cocaine.