I can't speak for the US, but in my country that was sometimes the case, but not always.

I was an engineer and had 'non-camo' uniform, and only wore camo in the field or on exercise.

People who's job was primarily 'in the field' though (e.g. the Army) didn't have that, and would wear camo as their uniform 90% of the time (the exception was formal dress uniform for parades etc.)

BDUs are working dress, comfortable and practical and easy to maintain.

Shirts and ties and pressed trousers look sharp at the start of the day but are much less practical and look creased and worn by the end of a shift.