> So when you need to figure out if a device is in the same building as you, is it easier to say "1.2.3. - oh, that's my building" or "1234:5678:90ab:cdef:1234:5678:90ab:: - oh, that's my building"?

This is a very poor straw man.

In IPv6 world you wouldn’t use all the significant digits randomly and because you have so many you could actually use one of the octets to perfectly encode the building information:

Building 1: 2001:1:: Building 2: 2001:2::

(You can go further with this concept and encode region/country/state/etc into the addressing as required)