From the US/Israeli perspective, it's hard to trust Iran, which is positioning itself as a threshold nuclear state, and may attempt to secretly cross that threshold at any time (as North Korea did).
From the Iranian perspective, the government is fearful of infiltration and overthrow, as well as outright invasion, remembering that non-nuclear states have been undermined by the USA and allies (as happened in Libya).