Wait, what? Why would anyone challenge them on having that power? That's like someone challenging Congress's ability to change or undo a law. Was their primary argument in this case "NO TAKESIES BACKSIES!"?
If the Social Security Administration (an executive agency) gives you $1,000 per month, it can probably decide to stop that monthly payment. But to get the money back, it probably has to sue you in an Article III court.
Conventionally, you need a court with an Article III judge for that to ensure due process.