Saying you need more data is like saying you need to flip a quarter 500 million times to get a better percentage estimate of heads vs. tails compares to 1 million coin flips. After a certain point, having more data only helps with identifying outliers and changes in behavior over time (when dealing with human/natural data).
The current state of the art of ML assumes nonlinear relationships between all parameters. It can't assume simpler & reasonable models, and therefore it can't extrapolate easily with reduced data.
I'm not really sure what "low-dimensional intuition" means, but I pretty regularly build models that do not "assume nonlinear relationships between all parameters".
https://en.m.wikipedia.org/wiki/Curse_of_dimensionality
The common theme of these problems is that when the dimensionality increases, the volume of the space increases so fast that the available data become sparse. This sparsity is problematic for any method that requires statistical significance. In order to obtain a statistically sound and reliable result, the amount of data needed to support the result often grows exponentially with the dimensionality. Also, organizing and searching data often relies on detecting areas where objects form groups with similar properties; in high dimensional data, however, all objects appear to be sparse and dissimilar in many ways, which prevents common data organization strategies from being efficient.
I can give you a beautiful and concise explanation of clouds too: as we all know objects fall back to earth, or more properly speaking they follow orbits, close to the surface they seem to follow parabolas, but in fact they follow ellipses. the moon orbits the earth in a nearly circular ellipse, as do geosynchroneous satellites. When water evaporates from the ocean they are in fact launched into an ellipcitcal orbit, however each elliptical orbit that intersects the surface of the earth will intersect it again, this explains rain, the individual water drops fly for a while until they fall back down...
beautiful, and relatively concise in comparison with real cloud physics, but totally wrong of course...
I have seen such insinuation multiple times, that samples "collect" near the unit d-sphere when samples are drawn from the unit d-sphere in high d dimensions.
From a physics perspective this is very familiar to me, but not in the sense of fact, but in the sense of misinterpretation.
I do believe the observation is very useful in the educational sense as long as it is pointed out as being a paradoxical illusion. In this sense I can appreciate (even encourage) a professor or a TA showing this phenomenon, on the condition they finish up by explaining why this seems to be the case, but is nevertheless a misinterpretation, and they should make sure the connection with Jacobian determinants etc are made clear.
Consider a normal distribution of any dimension (as high as you want), but I will showcase the phenomenon even with low dimensions (here merely d=3) to illustrate this has nothing to do with high dimensions.
Clearly the probability density is maximal in the center of the distribution.
In computer processing of data points, we typically loop over points, calculate some hopefully interesting function on each sample, and then plot the samples say by binning with equal bin sizes. A programmer typically disregards transformation properties like the Jacobian determinant. Suppose the value we calculate for each data point is the absolute length or distance from the center. The further we go from the center the smaller the probability density of the normal distribution becomes... but the larger the volume of a shell of radius r!
Since we are binning with equal bin sizes (equal length intervals per bin), for small lengths, then even though the actual probability density is highest near the center, we will get relatively few samples because the volume under consideration is small compared to the volume under consideration for a shell of a larger radius of equal thickness (area of a sphere grows quadratically with radius). However for even larger distances, the exponential decay of the normal distribution dominates and the number of samples in highest radius bins will decrease again. So in between there will be a peak.
This explains the fact that [ the probability density of [ the absolute distance from the center over [ the sample points ] ] has a peak at some non-zero length.
But it is a conceptual mistake to interpret this as if those sample points in the original d-dimensional space form a dense shell on some "unit sphere" ... This is a pure illusion which illustrates the interpreter is not familiar with jacobian determinants etcetera.
Consider volume or triple integrals over some volume element dx dy dz and for symmetry you prefer integrating in a spherically symmetric coordinate system, theta,phi,r then you can not simply replace dx dy dz with dtheta dphi dr, you need to use dV = dx dy dz = r^2 sin( phi ) dtheta dphi dr.
It is this necessary factor that is ignored when processing sample by sample and causing this illusion in the AI community. I did not follow conventional machine learning courses, but given the learned language used whenever I see statements to the effect of samples lying near the unit sphere in high dimensions, I can only conclude it has its origins in 1) direct observation or experience of plotting in bins of the length of the vector without guidance in interpretation; or 2) guidance during education, where the phenomenon is shown, and the origin of the paradoxical illusion or confusion adequately explained and then the illusory nature subsequently forgotten or 3) a teachers assistant having gone through 2) and showing the phenomenon to students without emphasizing the illusory nature of the misinterpretation.
But perhaps I am wrong, and the normal distribution in high dimensions actually has a higher probability density near its "unit sphere" if the dimension is beyond some critical dimension d_c... but again, I'd like to see a derivation showing it :)
EDIT: For more precise language: they do "collect" (reach a peak) at a certain non-zero length or distance, but they do not collect to a unit sphere of such radius in the original space!
There are also descriptions of the curse that do not involve spatial analogies at all. Assume that the data is independently identically distributed along each dimension and is an outlier if it's sufficiently far along one dimension, which happens with probability p in the one-dimensional case. Then in n dimensions, the proportion of outliers is 1 - (1 - p)^n -> 1 for n to infinity. Most points are outliers along at least one dimension.
I DO argue against the "collect near the unit d-sphere", if it does not come with an explicit pointer or reference to the explanation of this illusion, I don't care if one points to Maxwell Boltzmann speeds vs velocity vector distribution, or Jacobian determinant, but one should point to something else communication makes no sense. We communicate to teacch and learn. Only when explaining why there appears to be a sphere in the higher dimension, and how this illusion arises is communicating about this pseudosphere justified. The unit d-sphere makes no sense on the 1-dimensional axis of absolute length on which we project the samples. A reference to a "unit d-sphere" only makes sense as residing in the original d-dimensional sample space. But in that space there is absolutely no packing of samples near the peak radius as it appears in the distribution of lengths.
I was not responding to the "curse of dimensionality". I show that this effect already exists at low dimensions, and physicists are very acquainted with it because within their first years of university study they get drilled in 1) jacobians for non-linear coordinate transformations 2) the velocity distribution of molecules in the kinetic theory of gases, where there is a similar plot for the absolute speed (Maxwell-Boltzmann) distribution [0] showing a peak at a non-zero speed, even though the velocity vector distribution is a normal distribution... Every physicist worth his / her salt, immediately recognizes the phenomenon as relating to the Jacobian determinant, that this has nothing to do with velocities aggregating to some sphere of non-zero radius, and that this is a misinterpretation of magnitude distribution plot...
Clearly in this physics example d=3. Would you say d=3 already shows the curse of dimensionality? I call bullocks, and suspect a misinterpretation of the magnitude plot...
Again the peak in the magnitude plot is very real, any reference to a sphere with radius of the peak in the magnitude plot and residing in the original d-dimensional space is purely imaginary!
Of course the person with simultaneously avera height, avg weight, avg income, avg capital, avg age, avg ... is very rare, ... but less rare than a similarily accurately specified person with non-average weight, but all other variables still average...
[0] https://en.wikipedia.org/wiki/Maxwell-Boltzmann_distribution...
see the last 2 sections on velocity vector distribution, and speed distribution...
you seem to be referencing the fact that repeatedly convolving uniform distributions approach a gaussian distribution
One of the more extreme examples of this situation is Big Data, where everything is a convenience sample, and therefore nothing is particularly trustworthy. Combine it with standard statistical hypothesis testing, whose p-values, being designed for the size of experiment that one could realistically manage in a lab setting circa 1910, are as much a reflection of your sample size as of anything else, and you're at great risk of leading yourself into a tar pit.
Now imagine that you want to measure that bias across different types of coins and year of minting. Does the bias between coins of the same type vary more than across types?
Curse of dimensionality starts to raise its ugly head.