>> Judges are still allowed to defer to experts
> Then why is congress be denied the same latitude (via agencies)?
Come on, that's obviously false. Even in an alternative world where agency rule-making was totally abolished, and everything has to be enacted directly through legislation, Congress could still defer to experts to draft that legislation.
Also the courts themselves are supposed to be the experts at interpreting legislation. In our system of government, it's their job to have the final say about that.
Let me put it another way: do you think it would be a good idea for judges to be required to defer to prosecutors (also in the executive branch) on the interpretation of criminal law? (e.g. "Prosecutor: Your honor, the defendant is guilty because I say the law says he is. Judge: OK, your call. How long should we lock him up for?")