Posted On: Jun 8, 2020
Amazon Elasticsearch Service now offers anomaly detection, which uses machine learning to detect anomalies on streaming data and identifies issues as they evolve in real time so that you can mitigate issues immediately. This new feature is built on Random Cut Forests (RCF), a proven algorithm for real-time streaming, and is domain agnostic, making it a great choice for a wide range of log analytics applications.
Static, rule-based analytics approaches struggle to adapt to dynamic workloads and are prone to miss critical issues. Amazon Elasticsearch Service anomaly detection leverages RCF, an unsupervised algorithm that continuously adapts to evolving data patterns. The anomaly detection feature is designed to be lightweight and resilient, with the computational load distributed across Elasticsearch nodes, eliminating the need for dedicated machine learning nodes. This efficient design allows the feature to scale to handle large volumes of data without affecting cluster performance or application workloads.
The new anomaly detection feature includes a Kibana user interface. This UI provides context into the data and events that contributed to an anomaly, making it easy for all users, regardless of their machine learning knowledge, to derive value from the feature. You can use anomaly detection with alerting to trigger notifications, as well.
The real-time anomaly detection feature is powered by Open Distro for Elasticsearch, an Apache 2.0-licensed distribution of Elasticsearch. To learn more about Open Distro for Elasticsearch and its anomaly detection plugin, visit the project website.
Anomaly detection is available on domains running Elasticsearch 7.4 and higher at no additional cost. To learn more, see the documentation.
Real-time anomaly detection in Amazon Elasticsearch Service is available in Amazon Web Services China (Beijing) Region, Operated by Sinnet, and the Amazon Web Services China (Ningxia) Region, Operated by NWCD