4 extra bytes for a pointer is wasteful, but having an entire separate vector CPU just to make the windows transparent isn't?
Look at the big picture, whether it be pointers or integers into an array: the size of the index grows with the log of the number of things being indexed. You can't avoid this growth, but it matters less and less as you grow.