In general you are correct, but there are still easy cases where reversing is trivial.

For example if the shared output is the average (AVG = (X1 + X2) / 2). If I know X1 and AVG, I can easily calculate X2 (X2 = AVG * 2 - X1).

However, usually you want to calculate something more interesting, that depends on more variables, so you won't get an exact number on the other parties secret inputs.