Amazon Glue Data Quality now supports anomaly detection and writing results to the Amazon Glue Data Catalog
Amazon Glue Data Quality now supports anomaly detection for Catalog-based data quality evaluations and the ability to write evaluation results to Amazon Glue Data Catalog (GDC) tables. These capabilities work across both ETL jobs and Catalog evaluations, giving you a consistent data quality experience regardless of workflow type.
With anomaly detection support for GDC, you can identify unexpected changes in data statistics such as sudden drops in distinct values or row count spikes in your GDC tables using ML-powered time-series forecasting, without writing rules with explicit thresholds. This is especially valuable for data engineers monitoring hundreds of tables in the GDC who need to surface issues automatically.
With results storage in GDC, Data Quality rule outcomes, profiling metrics, and anomaly predictions (with confidence bounds) are written back to GDC tables, creating a queryable record of all quality evaluations. Whether the evaluation runs in an ETL job or directly on a Catalog table, results can be queried at any time using standard SQL.
Amazon Glue Data Quality anomaly detection and Catalog results storage are available in the Amazon Web Services China (Beijing) Region, operated by Sinnet, and the Amazon Web Services China (Ningxia) Region, operated by NWCD.
To get started, visit the Amazon Glue Data Quality documentation.