DeepLog: Anomaly Detection and Diagnosis from System Logs Through Deep Learning [pdf]

Anomaly detection is performed at two levels:

1. A log key anomaly detection LSTM model (a type of recurrent neural network) predicts the next log message type (log key). If it is wrong for any log entry, the operator is alerted that there is an anomaly in the execution flow.

2. If there is no anomaly in the execution flow, the parameter value and performance anomaly detection model is invoked. If the predicted parameter vector for that particular log key is very different from the actual parameter vector, the operator is alerted. The second model performs regression, in high-dimension space of the parameter feature vector. Therefore, the mean-squared error (MSE) is used as the loss function when training this model and when deciding to alert the operator.