That's not really true. If they could be added or removed without changing the sentence, then you wouldn't be suggesting their removal.

"To be honest" and "in fact" do change the meaning of the sentence, just in subtle ways. The question isn't a black and white "add meaning" versus "doesn't add meaning", the question is what is the actual effect of using those idioms.

"Adds meaning that distracts from the point you ostensibly want to make" would be more accurate.

Yes, although, I do find that those words do help make certain statements more convincing to those that read them -- without need for evidence.

Only add meaningful words. A "to be honest" or "in fact" can be removed. Brevity!