That's odd, I've been using i3 for years now and modals always open as a floating window. Even GIMP (and its many windows) works fine. I don't remember tweaking anything in particular to make it work.
My current setup uses xrandr to scale my 4k to 1440p.
It's not perfect, but it's better than other options.
It supports fractional scaling and different scales for different attached screens.. You have to tell the system you want it (`swaymsg 'output <x> scale 1.5'`) but it works.
You still need Xwayland installed for some stuff though. Most tools have adequate wayland replacements.