Another strategy I considered was using an infrared photodiode instead of a visible light photodiode. That way, the diode effectively reads the toast temperature rather than color, which would address the issues you mention. But I was a bit worried that the infrared emission of the heating coils would swamp that of the toast.

For my original design, you'd definitely have to set the control to only slightly darker than your rye bread, or you'd get burned rye toast.