The problem is that XUL was a) nonstandard and b) pretty badly designed and implemented. (For example: XUL trees relied on Javascript callbacks being called at paint time in horribly unsafe ways; XUL flexbox never integrated well with CSS block layout.) It was mostly hacked together in a hurry to get Netscape 6 out the door (and we know how that went). It was XML-centric and the Web ended up going in a different direction. All in all, it was clear pretty early on that XUL was going to be a dead end. Most of the good parts of XUL were standardized as CSS features (or in a few cases, HTML features).

Given more resources, Mozilla absolutely could have done a better job of transitioning faster from XUL to standards-friendly equivalents, and popularizing the latter in an Electron-like framework. But there was never a time when Mozilla had excess resources floating around, and other things took priority. (Also see my comment above about how architectural churn made it unattractive to support a stable embedding API.) As was pointed out in other comments, the benefits of regular Web market share tend to outweigh the benefits of embedding popularity.

XULRunner was working great.

Unfortunately, there were strategic decisions to basically never maintain it :(