Sites might use IP address information to guess that the sessions are correlated. But they can do that whether or not the connection remains open.
Responding to OP's "whether or not the connection remains open". IFF it remains open, then it's no longer a matter of correlating.
Imagine this scenario:
1. User visits a site in normal mode. Browser opens connection A. Browser uses connection A to fetch the site. Browser leaves connection A open.
2. User opens private mode. User visits the same site. Browser opens connection B to fetch the site.
How does it matter that connection A was left open if some other connection was used to fetch the site in private mode? If that's not the case, then its a big deal - but that requires some sort of evidence and not just wild hand waving that it could be the case.