i.e: A programmer knows how to do X, leaves a company to do Y, where Y is in the same field of work as X. Doesn't X still affect the programmer on a subconscious level and henceforth, their thoughts indirectly?
For various reasons (notably, the fact that slavery is illegal), we don't accept this in general. You have to show that secrets were copied in full. Employees cannot memorize millions of lines of source, they can only memorize vague architectural details that would be easily reverse engineered by competitors. If you want to own those vague details, get a patent, or shut up.
Wine does not allow people who have worked on Windows to contribute precisely for this reason, with some nuances as noted in the link above.
Venetian glassmakers were banned from leaving Venice with their technical know-how. The Venetian government employed assassins to take them down if they tried to defect.
Companies want your knowledge, not you bringing proprietary IP from their competitor to work, as they know that's a very expensive lawsuit waiting to happen.
How or why is the mere transport method of information the distinction between infringement of IP vs not?
Transport method it is because it means there's hard proof and a paper trail of the IP theft that you can prove in court. Memorizing something has no paper trail and also leads to a clean room deisgn. That's why companies want to hire people or entire teams, to do clean room deign, without any of the original data.
If I go watch a movie and then tell people what I saw it's fine. If I record the movie with a camera and show it on screen that's the IP theft.
We live in an imperfect world and solve problems as best we can.