The Fat man problem is just one variation of the trolley problem. It can be done with levers, instead. "Pull the lever, it goes this way".
This is what I find absurd - just because the questioner stipulates that two actions would have the same results doesn't make them morally equivalent (from the perspective of a person who can't see the future).
In the switch varitation both the 1 man and 5 man group are tied to the tracks. For whatever reason we can assume fate has brought them there and it is our role to simply make the logical choice.
But to ask one to push a man right beside you, in the same situation you are in, is quite different. Not even morally related.
The difference between the train track and the fat man is that the train track is obviously binary. Then the fat man problem contains words like "and pushing him is the only way."
But that's not at all convincing. You can't just say it's the only way. You'd have to rig up a situation where evil people set up a Saw the Movie type of hostage situation. In that case it would be just as bad for the acting hostage to kill 5 people in either scenario.