EXWM is the natural way to do this, treating X11 windows as Emacs buffers is what it's all about. Or are you trying to get just the buffer switching part without going all the way to EXWM?
By default, StumpWM lists normal buffers on C-x b, and X-windows using some other keybinding like C-t b