See Michael Nielsen's answer elsewhere in the comments for why this is not the case.
Another question is why L3-L3' red beam - the image info carrying beam - is fed into NL2 crystal thus providing a chance to physically affect formation of NL2-D5-L6 yellow beam, ie. to exchange information.
The interference requires us to not know which path the photon took way back at BS1, in that diagram. The probability of pair formation in the nonlinear crystals is quite low, so with very high likelihood, only one of the crystals will be responsible for producing a pair of photons (including the yellow signal photon.)
Suppose you did not route the imaging photon through NL2. Then observing an output photon at D3 would tell you that the pair production happened at NL2 (since there is no way for the imaging photon to get there.) This will destroy the interference pattern of the "yellow" photons. That is why the beam is sent through NL2.