X forwarding is just a tunnel for the normal X protocol, which is both higher-level and provides much more functionality than a raw pixel buffer.