If you don't keep it as a toy, it can get out of hand quickly. For example, here's the partial implementation of QWidget, almost 500k of code: https://github.com/qt/qtbase/blob/dev/src/widgets/kernel/qwi...
UI frameworks are one place where outside of toy projects, YAGNI doesn't really apply as much, and so things like focus systems and accessibility affordances should be factored in from day one. If you go in with the attitude of writing something "elegant", you're going to end up bolting these things on after the fact and writing contorted code to make it all fit together. It's like finding out that you need a skyscraper after building foundations and framing for a cottage.
You can avoid this complexity in the base widget class by placing those elsewhere, e.g. FLTK's base widget class[0] is much smaller at around 500 lines of code (+ documentation comments).
[0] https://github.com/fltk/fltk/blob/master/src/Fl_Widget.cxx