What is "data leakage"? Do the authors define it? They reference Kaufman et al, but that makes it sound just like "errors". But what are the errors?
Oh well, I guess this is only meant for insiders who understand the in-group jargon. The rest of us need not be interested at all.
Edit: for context, here is the paragraph titled "Leakage" from the draft paper:
Leakage. Data leakage has long been recognized as a lead- ing cause of errors in ML applications (Nisbet et al., 2009). In formative work on leakage, Kaufman et al. (2012) provide an overview of different types of errors and give several rec- ommendations for mitigating these errors. Since this paper was published, the ML community has investigated leak- age in several engineering applications and modeling com- petitions (Fraser, 2016; Ghani et al., 2020; Becker, 2018; Brownlee, 2016; Collins-Thompson). However, leakage oc- curring in ML-based science has not been comprehensively investigated. As a result, mitigations for data leakage in scientific applications of ML remain understudied
https://arxiv.org/pdf/2207.07048.pdf
Yes, alright. But what the flying fuck is "leakage"? Am I supposed to go read the "formative work on leakage" cited? What if that work also leaves it undefined and points to an earlier source? What the hell is this paper about? Hhow hard it is to explain what your main subject is, so I can read your paper knowing what you're talking about?
How frustrating.
There is a whole one and a half page section (2.4. Towards a solution: A taxonomy of data leakage) that describes every kind of leakage the authors considered for this work.
This is what will happen to most people who read the draft paper. They will start with the abstract, feel confused, get annoyed and leave before they get to the "texonomy".
In fact I did glance at the "taxonomy" as I was scanning through the paper dejectedly in the last few seconds I bothered with it and it still didn't seem to explain what the different kinds of data leakage are different kinds of. That's basically when I stopped reading.
And this is why you don't write papers like that, because it makes them less likely to be read. And because, as an author in a field where people put tens of thousands of preprints on the internet, you want to make absolutely sure that your paper is as likely to be read as possible, by as many people as possible.
Basically, if you want people to read your papers, the absolute worst thing you can do is to expect your readers to "keep reading to see what we mean". Most people will give up and reviewers will be so annoyed you're not respecting their time that they'll skim your paper looking for reasons to skewer it.
The next worst thing is to expect your readers to try and guess what you mean. I could very easily make an educated guess at what "data leakage" means, but then I would have to read the paper while never being sure I know what exactly I'm reading and whether I have it subtly wrong. Again, you don't want to cause that kind of stress to your readers. You want everyone who reads your paper to feel happy and calm and admire your bright, squeaky clean ideas.
So make things as clear as possible, as hassle-free as possible, as friction-free as possible. Otherwise, you're kicking yourself in the butt.
That's my free advice.