For me it's the debug experience that is lacking when only using containers, I have some work arounds but they are still not good enough.
I don't experience any difference, whether I am inside a container or on the host os.
I did have some issues with some projects and want to revisit this soon.