Agreed, though it could be one of those things where Fourier weirdness and neural network weirdness interact in counterintuitive ways. It'd be easy enough to perform the same experiment with some benchmark image classification dataset, do it with a few and you'd have good context for evaluating whether Figure 1 is an indication of leakage or not.