> “But why doesn't it feel like all colors?”
Because RGB is not a perceptual color space.
With RGB, a lot of values are wasted encoding colors that are perceptually very similar. Many colors that are “interesting” to humans are encoded in a narrow slice of the RGB cube. Much of the cube is a deep ocean of indistinguishable bluish shades.
Video traditionally uses various luminance+chroma color spaces, which are more perpetually effective than RGB and allow for easy subsampling to save bandwidth on the less important color data.