Yeah I think the nature of research is de facto searching in the dark. I mean if it weren't, would it really be research? I think of the analogy as the knowledge space is a dark void and our collective knowledge is permanent torches lighting up the area. Researchers are exploring into the darkness and finding suitable places to place the next torch. Maybe we can say that the torch's brightness is defined by how well known your work is, since people tend to the flames more (not to be confused with how important the work is).
The big question to me is about the geometry of that knowledge space. Is it bounded? If bounded (most likely imo), how do we approach the bound? Do we plateau once we hit it (unlikely imo)? Do we approach {,sub,super}-linearly? Do we approach {,sub,super}-linearly and then transition into an asymptotic convergence? (e.g. S-curve) This question actually has a profound impact on the question of the risk of a super-intelligence. IMO knowledge is most likely bound ( and most likely looks like an S-curve. If we're more than half way through that S-curve than a super intelligence that is 100x smarter than us may only know <100x as much as us, which we have good reason to believe that this is the case since knowledge appears to compound and accelerate (linear and sublinear models don't fit our historic trend, but that doesn't tell us the shape of the curve ahead). That reduces the risk of a super intelligence absolutely outsmarting us since it would be easier to close the gap, especially since learning through observation is easier than learning through exploration (can use its torches, even if they are more difficult to see). I'm not sure I see this really discussed much in the super intelligence conversations and that we're often working with different priors so that creates fundamental disagreements that become impassible. Maybe we need to set these priors first before we even discuss the next priors about motivation of a super intelligence.
At least for math, I don't think I have the capacity to learn it all in the short human life, even if we're just discussing what math would be helpful to ML. But it sure is fun to learn it, so I guess the inability to learn it all is not really a downside :) Now if only I can convince others that math is fun __and__ useful to ML.