The capital city of the Incan empire, Cusco, is estimated to have had a population of ~40,000 [1] ruling over an estimated 10,000,000 people.
The capital city of Spain at the time, Toledo, had a population of a mere ~32,000 [2] at the time ruling a population of a mere 6.5 million at the time.
Contemporaneous accounts demonstrate the comparable grandeur of their seats of power and more objective determinations demonstrate the comparability of scale and scope. It is not just “a handful of fancy residences” unless you apply that moniker to Spain as well.
For that matter, Spain in the early 1500s is not exactly a wealthy country. Their wealth largely came from their American conquests which resulted in what is believed to be a ~300-600% increase in the amount of European-accessible gold and silver [4]. It is a little hard to determine how much came from plunder and how much from the mines and slavery, but the Incans were not poor in precious metals, the measure of wealth in the contemporaneous European societies.
[1] https://www.worldhistory.org/Cuzco/
[2] https://en.m.wikipedia.org/wiki/Toledo,_Spain