The client that I am using (which you can download from https://products.office.com/en-us/skype-for-business/downloa...) keeps all conversations in one window with people listed down the left. I've never seen it starting new windows.
Which client? The OSX client appears to have very limited configuration options.