Combining Deep Learning with Pattern Discovery for log monitoring

It is fair to say that pretty much any organisation in IT field deals with logs or metrics. There are different types of them, some pretty well known (like e.g. Apache logs). However,  there is also a long-tail of the less popular (or custom) services producing data, for which no parsing rules actually exist. In this talk we will present a novel real-time pattern discovery method that automatically rebuilds the latent log structure, allowing to think about them in terms of an event (pattern) and its properties (parameters). This makes it an excellent input source for Deep Learning Anomaly Detection algorithms. Combined with powerful UI, the user is given with tools that provide insights on the most important events quickly. The talk will present two specific use-cases in security and performance fields where such approach dramatically raised awareness on the important things happening in the monitored environment.