The office of the Director of Customs wrote six letters requesting removal over several years, but didn't get a response. It looks like the Port Authority was responsible but there may be court orders involved.
We now know how the ammonium nitrate got there — a Russian ship bound for Mozambique which ran out of money and got stranded — but we don't yet know exactly why it was never removed.