Is there any good reason why a fully-connected network needs more than one hidden layer? Theoretically, any non-linear function could be mapped into a fqn with only one hidden layer. Does deep has anything to do with fqns or only with cnns?
A sufficiently (infinitely?) wide shallow fcnn can work in theory, but is basically impossible to train[0].
It might be feasible to transform or 'compile' a trained deep network into a shallow & wide one, but I'm not sure there would be any benefits, absent sufficiently wide parallel hardware[1]
[0] Then again, deep networks were impossible to train for quite a while too, due to the exploding gradient problem.
[1] Although, the Cerebras Wafer-Scale chip does exist. Hmm.