Not as such, but larger caches are slower (due to the speed of light) which can reduce performance. A few apps have seen performance regressions on Tiger Lake.
Wouldn't it be more accurate to say due to gate delay because of an additional level in (de)muxing?