I'd recommend two things. The first is Brad Myers' Software Structures for User Interfaces course at Carnegie Mellon University, which focuses on the guts of how graphical user interfaces work.
https://www.cs.cmu.edu/~bam/uicourse/05631fall2021/
The second is Dan Olsen's book Developing User Interfaces, which has all of the details of how GUIs work, from graphics to interactor trees to events to dispatching. For some reason, it's absurdly expensive on Amazon right now.
Both Dan Olsen and Brad Myers were early pioneers in GUIs and GUI tools, so you'd be learning from the masters.