You could expose enough GTK bits to expose an event loop to the LGI lua library. It's gobject-introspection for Lua. Since you already use these libs, it would not make Ardour any bigger.
I am not saying it's a great idea to mix GUI and a realtime DSP in the same thread, but it would be supported in you see some demand there.
https://discourse.ardour.org/t/is-open-source-a-diversion-fr...
There's really no technological reason for not allowing Lua to create GUIs within Ardour. It's more a question of whether or not we actually want to. Either way, you would not be mixing GUI and realtime code - the architecture of Ardour doesn't allow that.
The question I was raising (which I think you understand) is whether most users care that this is possible if it can't be done without a rebuild (compiling).