From the first chapter of this book, it is described that each layer adds more abstraction. The first layer (input) works on individual pixels, the next layer on a part of the image. Each deeper layer can make more high-level decissions.
There's probably room for experimenting with more novel constructions and even some papers out there looking into the matter.
Although if you wanted to come up with difficult scenarios, it's not hard to think of structures that would make some of those middle layers really tall, or add a lot of middle layers.