Wow I never thought about Ctrl+Click to deselect in that situation. I've probably wasted at least an hour of my life by giving up and re-opening the window in a different way. Thanks!
(I actually asked someone more knowledgable about this once and was told that it's my fault for using the file picker incorrectly and I simply need to stop doing that.)