And let's not forget the accessibility systems, clipboard handling, filesystem (e.g. a GUI toolkit needs file open/save dialogs, which means you want to monitor the filesystem for changes, which is different for all operating systems), audio, notification systems, printing, ... And probably many other things I can't think of right now which modern toolkits like Qt or GTK have to deal with.
A full GUI toolkit is already an incredibly ambitious endeavor and cross platform compatibility only makes it much more unrealistic to achieve in a timely manner.
A desktop environment on the other hand can be much less work and can grow more naturally. For example you can start with a window manager/compositor (which there are good libraries for), launcher and dock/panel. All the applications (file manager, image viewer, ...) can be leveraged from other DEs and replaced step by step if needed.