A while ago I looked into this for a similar-ish hobby project, and the main dealbreaker seemed to be the mandatory AC coupling capacitors: they are intended to block DC currents, so a signal which is substantially slower than intended is essentially fighting a high-pass filter. This is also why there are special AV SPF transceivers: Unlike Ethernet, SDI suffers from "pathological patterns" consisting of extremely long runs of 1s or 0s, which can cause "DC wander" [0]. SDI transceivers need to take this (albeit extremely unlikely) possibility into account, or risk losing signal lock.
For this reason I pretty much gave up on the idea of reliably going sub-100Mbps on cheap and easily available 1G / 10G SFP modules. Seeing it (mostly) work for TOSLINK at 3Mbps is beyond my wildest expectations - I bet the LVDS driver's high slew rate is doing quite a bit of work here too.