You're right. Just don't joke around about it. Privacy matters.
I wouldn't hesitate to reprimand and/or fire a system administrator that repeated (much less joked about) any personal information they came across in the course of their duties.
On the other hand, as an employee, I think it's also your responsibility to assume that any of your communications may be monitored by someone less polite than you would prefer, and to act accordingly.