The NeWS book at page 35 has a picture explaining the structure of the CMU Andrew Window System: the window manager on top of the "base window system" (assuming protocol and logic), which sat on the graphic library, and applications connected to it through sockets. A limit was that the window manager was fixed, so you couldn't change its policy/look and feel without having the sources and recompiling it (and here I wish the source code was available).
Wayland it's kinda similar, but most of the graphic layer is moved from the server into each application (where each one use its own). Unless you have a compositor that is designed to be extensible out of the box, a set of widgets will be needed if you don't ship a desktop environment, and each one is a separate system process (for example: toolbar, clipboard manager, status bar, wallpaper). Another limitation is its "static" protocol, in the sense that you cannot extend the compositor at runtime loading extensions (you have the XML which is used to generate the stubs for the implementation in the compositor AFAIK). This can prevent applications that use them to run on compositors that don't implement them.
On the other hand, systems like NeWS, GMW (a window system from Kyoto University, ASTEC and Omron) and the one described by Rob Pike in "A concurrent window system" are interesting for these reasons:
- runtime extension loading code: new functionalities to the server can be added when needed
- code as a protocol: instead of sending "raw data" and requests, equivalent pieces of code are sent, allowing for compression and to offload a portion of the logic to the server (or even have applications that run inside the server). This can also allow to have applications that run on remote machines, which download the code for the interface to the server.
- concurrency: their programming model is structured around "lightweight processes" (or equivalent), which allow to decompose the interaction in independent tasks, which can eventually interact by sending messages/events or sharing resources.
I think it's worth re-investigating this kind of design, considering also the technological advancement (like the evolution of graphic hardware, multicore CPUs, touchscreens, ecc) and challenges (including security). It would be interesting something like NeWS but a bit more abstract and capable of supporting whatever graphic model you wish.