Quartz Composer is a graph of filers that you can interconnect in a graphical interface. You use it to create visual filters or generators. iTunes was using Quartz composer files .qtz for some of his music visualizers.
Quartz Composer is GPU accelerated thanks to its use of the Quartz API.
It was very popular with artists because of the creative freedom it gave when composing the filters. You did not need to be a developer to create a filter, thanks to the editing application.
Quartz Composer is now deprecated and dying in slow and anonymous death.
XQuartz is a windowing system accelerated with Quartz. The X system was not invented by Apple but very popular on top of Unix.
xquartz does the X11 protocol and displays it on apple display.
quartz composer is a sort of graphical composition tool.
They just happen to have quartz 2d graphics api in common.