Yeah you have actually picked up on something that the game intentionally does to help the user: If the cheese can be reached by any of the dimensions you are currently on, then it will ensure that dimension is in the subset of dimensions you receive when you go down a dimension.
Otherwise I'd have to introduce some kind of dimension picker. Maybe a fun hardmode.