So, you can't necessarily do that because video is compression, and compression means not predictable. (If it's predictable it's not compressed well enough.)

That means you have to stick to inside the current block. But there are some tricks; like for an IDCT there's a previous stage where you can rearrange the output memory elements for free, so you can shuffle things as you need to fit them into vectors.