> It sounds like you would need to be exceptionally careful that your meta-process didn't "find" some signal in pure noise (via re-using test sets and so on).

It sounds like you’re actually talking about ordinary levels of carefulness in this (ML) context.