It shouldn’t, unless your phone is placed against a magnet. The earth’s spinning magnetic core is the size of the moon. Those nearby buildings will not interfere with that.
The phone may be using GPS for headings instead of an actual compass causing it to act that way.
Why would it do that? Digital magnetometers are cheap, low power, and ubiquitous.
Some inexpensive phones don't include it due to cost reasons, it could also be bad software or a defective magnetometer in the unit.
And also, IME, completely unreliable. The compass in my car's sat nav never points north, and my phone's compass constantly needs recalibrating.
> It shouldn’t, unless your phone is placed against a magnet.