That's because your lawyer would be able to point out all the ways in which the agreement was not a contract.
As we recently saw with the Zillow letter to McMansion Hell, a cease and desist letter is a legal document, but that does not necessarily mean that it has any inherent merit or correctness.
In order for a judge to be able to grant relief in case of a breach, the document has to meet all the criteria for a legally binding contract. If there's no contract, neither party has to do jack squat for the other.
There are exactly zero contracts existing between me and my current employer. We have an informal, unenforceable-in-a-court-of-law agreement that says I will do X and not do Y, and then the employer will pay me Z every two weeks. Any day of the week, I could show up to hear, "Our deal's off. Go away, and never come back." I'm not ecstatic about the arrangement, because there is a sizeable power imbalance between me and my employer, but I can't do anything to change that individually. But I do work and get paid for it.
All those agreements do is let me know that I shouldn't bother doing certain things, unless they would allow me to quit my job.