It’s simple. Pointer sizes being twice as big consume twice as much cache, thus you’re effectively cutting your cache size in half (worst case, if the cache line were stuffed end to end with pointers). I imagine the idea that cutting the cache size by up to 50% causes a severe performance impact doesn’t require much explanation.