One problem is termcap/terminfo itself. The definition has expanded greatly over time, and documentation is poor to non-existent. In many cases termcap/terminfo has been implicitly extended to support ncurses specifically. And like you say, everyone just sort of assumes ANSI/xterm -- nobody has `PS1="$(tput setaf 3)\w $(tput setaf 0) $", you just hard-code the codes.
The last time I chased the rabbit down this hole I got mired in TTY-land and was not able to dig myself out. Too many insane layers, and before you know it you're writing code to emulate serial connections in a vain attempt to make things work without burning the whole world down.
Both (n)vim and tmux have internal TTY emulation as well which makes things even crazier; I think vim uses libvterm and tmux has one internally hand-coded. It's a mess.