That depends on where your cache is.

If your cache is on a separate box and you have to make a network round trip, yeah it is expensive.

But if your cache is in-process, the access is cheap.

Also, CPU tasks like authenticating a message are cheap to scale horizontally. A cache is a centralized bottleneck.