The convolution kernels in the first layers of AlexNet and all its DL image processing descendants converge to the Gabor filters (or some variation of) which are the response functions of the neurons in the first layer of the visual cortex. About 15 years before AlexNet there were works showing that such type of filter is kind of mathematically optimally encoding for the feature based image processing. (So, theoretically one could have just pre-generated the first layers in the net and use them fixed thus cutting significant time/effort on the training - i myself wanted to do it 20 years ago, yet just didn't get to it :)
I'm pretty sure that for middle layers in image DL as well as for transformers in lang we have a kind of similar optimality, ie. something like maximum entropy filter(separator/aggregator at higher levels?) at a given level of granularity/scale like Gabors at the first feature level.