We primarily say "Well, actually..." to each other, and it's pretty much unavoidable. Physics is a diverse, complex, and difficult field. People working in any given niche have to make a plethora of simplifications and assumptions about peripheral stuff just to make it manageable. It's a bit like how computer scientists usually don't worry about the solid-state physics of a logic gate, how a CPU is designed, or how compilers are written when they're writing a program. Yes, it's the foundation their work is built upon but, most of the time, they can safely ignore it.
When we say "Well, actually...", some of us are letting our insecurities show by trying to one-up everyone else. Some of us (probably a lot of us actually) are obsessive about certain things that fall within our bailiwick. We have difficulty staying silent when somebody smooths over something wonderfully crinkly that we can't resist talking about.
Mostly, we're just pointing out things that we find genuinely interesting. This serves a valuable purpose. Sometimes the simplifications and assumptions we make bite us in our posteriors. When we talk to other physicists, mathematicians, etc. outside (or within) our own specialties, sometimes the "Well, actually..."'s lead to important realizations.
Most of the time, it doesn't really matter. Well, actually, sometimes it really does matter!