The type of reproducibility is different here. What you mention is possible via a stable compiler ABI already. However one needs to keep the source code the same. Without a stable compiler ABI, you may or may not fix it depending on what the compiler does.
The goal of reproducible builds is removing sources of environment-dependent behavior at the build level instead of the compiler level. So given all the same dependencies and same build commands your binaries should match wherever and whenever you compile them. The distros and software developers also made a huge effort to remove any kind of environment-dependent commands.
Different distros still have differences in the build commands they issue and the set of dependencies they enable. The space to cache each individual possible output would be enormous and impractical. So you would still need repositories.