I‘ve recently come to the conclusion that the magic of fully connected neural networks is that there are almost no tricks to reach close to sota. Dense layers + relu + adam = it just works
There are many rules of thumb that took the last 5+ years to discover but are now quite standard. You are nit picking on fully connected, but if we add dropout, weight initialization, and adaptive learning rate to what they said, then we are fairly close to being able at least get a deep architecture to overfit a toy dataset and be off to the races for then applying it to a larger dataset.