M1 Ultra bandwidth is for CPU and GPU (800GB/s). Grace is just the CPU. Hopper, the GPU, has it's own memory and bandwidth (3 TB/sec).

The M1 memory bandwidth is mostly for the GPU but Grace does not include an (on-chip) GPU.

The CPU complex on the M1 series doesn't have anything close to the full bandwidth to memory that the SoC has (like, half). The only thing that can drive the full bandwidth is the GPU.