Well, there's also observations from neuroscience. CNN's were directly inspired by observations of closely-grouped neurons firing when detecting certain images, and capsule networks are directly inspired by cortical columns.