So first, not all science requires an RCT. Dividing expiremental subjects into study and control groups is one way of doing science. It's not the only way.
In this case, this is a concrete demonstration of just how much variance can emerge from a "statistically neutral" process. The systemic flaws are part of the demonstration. What appear at first glance to be identical tools, inputs, and processes are in fact subtly different. The demo shows management types that their charts and graphs cannot always be relied upon to differentiate performance levels among staff. The system itself must also be scrutinized. If Bob's ad campaigns are outperforming Alice's by 20% in the first quarter, it doesn't necessarily mean Bob is a marketing genius and Alice needs a PIP.
A computer simulation would not have nearly as powerful effect on most people as a live demonstration using real beads. And the imperfections in the paddles is something that naturally arises when they're physically made, but would have to be tuned by the programmer building the simulation. Which would lead to questions about "just how did they decide what variances would come into play?"
Meanwhile another worker—previously scoring 5, and getting a merit-based raise from it—did poorly with 12. The remark: "That raise went to his head. He's getting lazy".
So yeah, the value of this is in the actual doing of it.
If you are curious (and have the stomach for some MBA-ness), this article (four parts) should be well worth your time: https://archive.is/tXJhw
This is not in their control — they’re victims of corporate policy.
Does it influence quality in complex and hard to quantify ways?
Most assuredly…
If that's his point, it seems obviously wrong. Some work, like in the experiment is dominated by system behavior. For others, system would play a much smaller role. For example, 2 people cranking code in a startup. No matter what system you apply, if they are not good programmers, nothing of value will come out.
Randomly swap in two new actors with different life experiences into the same spots to do the same work, and you'll still get shit. If in the unlikely event, you get amazing work, it's not that the people doing it were special; it's just anothe outlier in the data stream. Add in the emotional toll of working as hard as possible to succeed but never being able to meet prescribed quality levels?
A system is perfectly tuned to produce the results it does. Want different results? Change the system. That is Deming's point. We have a tendency to blame variance in a system on the human actors immediately proximal, instead of paying attention to the actual significant constraints. This is an important lesson to management types, as they are to process/system what a programmer is to a computer.
The planners cast the dice for downstream long before downstream can do anything about it, and in many corporate setups, top down works just fine, but bottom up never gets any attention.