Thanks for giving an excellent example. NuPRL is exactly like this, but no one in their right mind would say the group that put out NuPRL hasn't done good science for 20+ years.
I don't want to mix up the side issue of "should code-based research always provide source code" with the metaphors being discussed. I definitely think useful CS research can happen without source code needing to exist, which I hope is the NuPRL project's case.
I do feel code based research should provide source code that was created as part of the research when it is material to their claims, including 'we ran a simulation' and similar findings. If you already have that level of detail, you should include it. If you make claims based on data, you need to provide that data. If you make claims based on source code, you need to provide that source code.
Difficult to obtain and get running (I'm told by everyone I've asked). So definitely not the first thing; mostly those last two things.
And yeah, I think that high-quality source code is definitely a pretty impressive feat when researchers pull in off. And I also agree source code should be provided. But beyond providing a VM, I think it's really dangerous to sort of fixate on building software that's easy to setup/use. There's a pretty significant time cost there in some cases, and it's worth asking whether that's what we want to be spending our Science dollars on.