My point was that there is only so much we can understand. Let me give you a concrete example, one which I have chosen to be easy to understand--the irony!-- : You are a biologist and are given the task of reverting skin senescence in a billionaire client of yours. I'm choosing this example because senescence is a very individual process, with different biological pumps[^1] stopping at different points in time and for different reasons. You can choose to understand how the processes worked together to produce the present system state and skin condition. But that's not your task, your task is to revert it. Understanding seems like a logical first step, but along the way, you (always) discover that these processes involve tens of thousands of interactions between an order of magnitude more of metabolites working at different stages and compartments, and that you can't keep a general intuition of them in your mind[^5], other than the very basic "sh*t breaks". But that's okay. You can always put all of it in a database. Then you only need to remember where the database is, and the dozens of different simulations that are interacting with that database. You will also need to understand the organization of the database, and what the simulations are doing, but there are way less of those and they follow human-made ontologies, sometimes they even come with documentation. If you play with those toys correctly, you will come with an individual intervention for your billionaire that you know will be sound, even if you don't have a comprehensive chain of reasoning of why expressing your customer's variation of 3D9S[^2] 7.5% less will help make his skin better[^3].
In any case, this is an area where there is some vigorous debate[^4] right now.
This is somewhat similar to how we don't understand the precise effect of a weight amount trillions in a LLM, but we can still architect, build and profit from the LLM.
[^1] That's a name I use for clusters of connected pathways, but the distinction is arbitrary and in this case the clusters were created by a graph clustering algorithm.
[^2] https://www.rcsb.org/structure/3D9S
[^3] If you are thinking that I should have made this example about cancer: the most frequent cause of cancer is cellular senescence. I couldn't muster the cynicism of making an example about the symptom instead of the cause. But most of my colleagues in search of public funding will. Go figure.
[^4] https://direct.mit.edu/posc/article-abstract/31/5/594/115643...
[^5] Or, worse, you risk holding to the wrong intuition or understanding. Because we tend to misunderstand complex things much more easily than simple things, you know.