Firstly, I would like to thank everyone for taking the time to go through the paper and raising your questions. I think most of these are valid questions and I will try to answer them here.
First of all, regarding the dataset creation method and the chances of having overlapping patches between training and test sets - as we mentioned in the paper, care was taken to avoid this specific scenario where a homogeneous landcover area was first sampled and then 28x28 patches extracted from it occur both in training and test sets. For this, once we selected the set of 1500 tiles from the NAIP dataset, we separated the tiles into spatially/geographically distant areas and then used these two areas for extracting the training and test patches respectively.
Now, regarding the comparison with raw pixel based DBN and CNN. As benanne suggested that we claimed that DBN is better suited than CNN for this task, in contrast, I would like to clarify that in the paper, we claim that the integration of Haralick features with the DBN yields better results than either of DBN or CNN with raw pixels. In fact, validating the popular belief, we show that DBN alone is indeed inferior to CNN on this task same as what benanne states. Now, the question arises - why didn’t we use bigger CNNs with higher number of layers - may be 5 or 6 for the experiments? Here, I would like to point everyone to the paper “ImageNet Classification with Deep Convolutional Neural Networks” where the authors trained a deep CNN with 5 convolutional layers and 3 fully connected layers. But we can see that the dataset used in that paper consists of 256x256 images. As the theory of CNN suggests, a CNN models the human visual cortical system that builds a hierarchical model of the image in such a way that bigger images require deeper nets to encode all the contextual dependencies between neighboring pixels and to perform a hierarchical clustering from the features to labels. So, in contrast to their dataset, we see that the images in our dataset are only 28x28 which means that we need to scale down our model to avoid significant overfitting which can’t even be avoided with L2 regularization or Dropout. Also, if we consider results on another object recognition dataset, namely CIFAR-10, which is 32x32 (roughly equal in size to our images), most of the state-of-the-art results on this dataset (like Deeply Supervised nets and Network in Network) uses an architecture with 3 convolutional layers and 1 fully-connected layer same as the one we considered. Also, in the Imagenet classification paper, while using the significantly being and Deep networks, in order to avoid overfitting the authors use techniques like data augmentation and dropout. So, effectively the model would be bigger and take longer to train while the integration of the texture based features with the Deep Belief Network saves us this overhead which is significant for a 65 TB dataset (It should be noted that our goal in this research is not just to handle this dataset but to use this as a pilot to develop an algorithm that can scale for landcover classification to the whole of continental US). Now someone might ask, why we chose 28x28 in the first place? Its because an important difference between satellite image classification and object based classification is that given an image with a bigger context, we say if a particular object is present in a scene or not (for e.g., CIFAR, Imagenet etc.). On the other hand for satellite imagery, what we need is a near accurate per-pixel labelling. So, choosing a smaller window removes most of the contextual information required for a per-pixel scene classification and choosing a bigger window means loosing the statistical properties of the object of interest - e.g., a tree or bush might be much smaller than a 64x64 window which covers a spatial area of 64mx64m. To conclude, I would like to stress the fact that out of traditional DBN, CNN and SDAE, CNNs are probably the best suited to handle these kinds of datasets and that is also proved in the experiments in the paper. However, I would like to re-iterate that in this research we try to show that even if Deep Neural Networks - DBN, CNN or SDAE are good as general-purpose learning machines and are versatile to suit a large number of different datasets - ranging from handwritten digits to facial recognition to object recognition, but, there are situations where we need to integrate these models with domain specific knowledge to help improve the learning efficiency of the model in the light of the particular decision problem at hand.
Also, “inherently high variability” is not a problem if we are ready to use bigger networks which in turn require higher number of training samples (according to the theory of learning for neural nets) and higher training time. However, as I told earlier in order to improve the overall efficiency of the learning algorithm to be scalable across millions of scenes covering the whole of US, the integration of DBN with handcrafted features seem to be very useful at this point of time.