I’ve seen some that are basically build as two concrete block or brick walls, and the center are was filled with rubble stone and dirt.

This is similar to an “earth house” and is very comfortable where the average (or basement) temperature is comfortable. The walls make an insane amount of thermal mass and the temperature inside changes very slowly.

It's not insane. I currently live in a house where some older rooms have 0.9 meter adobe brick walls. These were built in mid-XX century, when this was normal. Now we have better materials and don't need that much width.

My family house was expanded so it had a thick wall like that between kitchen and the new part of the house. It was so thick it barely got wifi thru