Fear of having any record used against you by a third-party is a completely rational fear, since as you noted, regardless of your intent, simply viewing third-party IP that is publicly viewable, such as an existing patent, introduces potential claims of theft regardless of the facts.
On the flip side, if done correctly, such records can provide supporting evidence that the claimed conflict did not occur.
For similar case law, see: