You still may want to replace softmax layer with support vector machine for classification sometimes.