I can poop outdoors on backpacking trips using natural materials to wipe, I can poop in busy airports and I can poop in stinking 100 degree portapotties.
But I have a really hard time pooping in gleaming offices where all my coworkers can see me walking into the bathroom and emerging 10 minutes later. I hate it!
The answer to the latter, if you are not a horrible person, is of course not. You don't really care, it's not your problem, you're not keeping track of it, it's just an event you forget moments later. Probably, then, that's what your coworkers think of you, when you go; so stop caring, because they definitely don't.
(If anyone ever were to metaphorically give me shit for using the bathroom, I'd turn it around on them in a heartbeat, too. It's so easy - just ask, what kind of creep cares about their coworker's bodily functions to this extent? Gross, weird!)
before WFH, I worked at a desk in an open floorplan. for whatever reason, it gives me a constant low level of anxiety to be in an environment where lots of people can see me, especially if I can't easily tell when people are actually looking. so I used to go to the bathroom a bit more often than I really needed, just so I could spend a few minutes in a more private space. I personally don't care, but I certainly do notice which of my coworkers also seem to be frequent users of the bathroom. if I were a pointy-haired boss, that might be one of the data points used to assemble my list of slackers.