I think the best way to do this would be to use something deterministic like instruction counts for the actual pass/fail. You can include wall time for information.
Yes definitely, this is what we already do with cpu simulation. But we had many people with benchmarks using syscalls for network,fs and other resources. And we found the only solution for this is to actually measure wall time