TL;DR
Logs are valuable information describing runtime events. However, because of the sheer amount of logs generated by large online services, it is impossible to get any insights from them in real time using traditional observability tools. This is why at PacketAI, we built a real-time log analysis engine based on text mining and machine learning algorithms.
Since logs are mainly unstructured textual data, the first step towards automated analysis of logs is the “representation phase”, where raw log lines are transformed into structured data ready for downstream tasks such as anomaly detection. In this article, we present our design for Log2template, a universal log parsing engine that is agnostic to log formats. Log2template at its core, is based on word embedding using neural networks.