Yes, throughput would be larger and we would loose the backprop path like you said, but it seems practical in some ways and actually guiding approaches like https://nv-tlabs.github.io/meta-sim/