The trick is to behave like you're not doing anything wrong.

If you have a pair of bolt cutters and you look like you're aimlessly wandering, people will notice.

If instead, you walk in and go straight into any random office, you're fine.

If you know where the building manager's office is, go straight for that and make it look like you talked to them, people won't question anything you do afterwards. (Source: watched someone do this during a pentest)