Auto-remediate best practice deviations detected by Amazon Web Services Trusted Advisor

Amazon Web Services Trusted Advisor inspects your Amazon Web Services infrastructure and provides best practice recommendations when opportunities exist to reduce cost, optimize your Amazon Web Services infrastructure, improve system availability and performance, help close security gaps and monitor service quotas. Trusted Advisor recommendations are based on best practices identified by Amazon Web Services services experts and learnings from serving thousands of customers over time.

In this blog, we present an architecture pattern that you can implement to automatically remediate best practice deviations detected by Amazon Web Services Trusted Advisor. Auto remediation of the Amazon Web Services Trusted Advisor recommendations help:

• Streamline your operations,
• Respond to security vulnerabilities quickly
• Maintain an optimized Amazon Web Services environment.

Exposed IAM Access Keys pose a security risk to your Amazon Web Services accounts and users. Customers are ultimately responsible for the safety and security of access keys. In our example, we will show you a solution that automatically remediates the Exposed Access Keys security check.

Overview of solution

Here’s a general approach:

1. S ign in to the Trusted Advisor console
2. Identify security recommendation: Use Trusted Advisor to identify security recommendations in your Amazon Web Services infrastructure. Trusted Advisor checks for common security vulnerabilities, such as Exposed Access Keys , Amazon S3 Bucket Permissions , Security Groups with Unrestricted Access and many more. A detailed list of Trusted Advisor Recommendations is available here . If Amazon Web Services Security Hub is enabled in the Account, Trusted Advisor helps to identify potential security gaps reported in the Amazon Web Services Security Hub Foundational Security Best Practices .
3. Configure event-driven automation: Subscribe to Trusted Advisor events in Amazon EventBridge to receive notifications about events you intend to automatically remediate. Using an Amazon EventBridge rule, forward the event to an Amazon Web Services Lambda target .
4. Implement auto-remediation actions: Create a Lambda function as the target to respond to the event from Amazon EventBridge and execute remediation steps.
5. Notify security team: Ensure that your Lambda function logs the details of the remediation actions taken. Use Amazon Simple Notification Service (Amazon SNS) for sending notifications about the auto-remediation status.

Solution Architecture

The following  diagram illustrates the solution architecture to automatically remediate the Amazon Web Services Trusted Advisor security check Exposed Access Keys recommendation.

Figure 1: Solution architecture for automatic remediation

Solution Workflow

The solution workflow is composed of the following steps:

1. Trusted Advisor detects the publicly exposed key and sends an Exposed Access Keys security event to the default event bus in EventBridge. The structure of the Exposed Access Key event looks like the following:

{
  "version": "0",
  "id": "1234abcd-ab12-123a-123a-1234567890ab",
  "detail-type": "Trusted Advisor Check Item Refresh Notification",
  "source": "aws.trustedadvisor",
  "account": "123456789012",
  "time": "2018-01-12T19:38:24Z",
  "region": "us-east-1",
  "resources": [],
  "detail": {
    "check-name": "Exposed Access Keys",
    "check-item-detail": {
      "Case ID": "12345678-1234-1234-abcd-1234567890ab",
      "Usage (USD per Day)": "0",
      "User Name (IAM or Root)": "my-username",
      "Deadline": "1440453299248",
      "Access Key ID": "AKIAIOSFODNN7EXAMPLE",
      "Time Updated": "1440021299248",
      "Fraud Type": "Exposed",
      "Location": "www.example.com"
    },
    "status": "ERROR",
    "resource_id": "",
    "uuid": "aa12345f-55c7-498e-b7ac-123456781234"
  }
}

To monitor Amazon Web Services Trusted Advisor check results with Amazon EventBridge, please check the product documentation here .

2. A rule created in Amazon EventBridge filters out Amazon Web Services Trusted Advisor events matching the Exposed Access Keys event as shown in step 1 above. It then triggers Amazon Web Services Step Functions to initiate the remediation workflow.

3. The step function workflow remediates the security exposure by invoking three different Lambda functions, each accomplishing a specific task.

4. The Lambda functions perform following tasks

4.1 Lambda Function: delete_access_key_pair: This function disables or deletes the exposed access key pair in IAM.
4.2. Lambda Function: lookup_cloud_trail_events: This function queries Amazon Web Services CloudTrail logs to fetch the most recent API invocations linked to the user associated with the exposed access keys. The default lookup interval is set to 24 hours, but it can be adjusted as per user preference.
4.3. Lambda Function: notify_security: This function sends a notification to the recipient about the remediation action taken. It includes the data retrieved from CloudTrail logs by the Lambda function “lookup_cloud_trail_events”. Alerts can be sent to a pre-configured Amazon SNS topic with an email subscription . The alert destination and recipients can be customized by the user according to their needs.

Technical Architecture Implementation

This section explains how to implement the architecture in a single Amazon Web Services account, or across multiple accounts within Amazon Web Services Organization.

Deploying the solution in a single Amazon Web Services account

The prerequisites to deploy this solution are:

1. Trusted Advisor should be enabled in the account.

2. The account must have a Business, Enterprise On-Ramp, or Enterprise Support plan to create EventBridge rules for Trusted Advisor Recommendations. You can verify your Account Support plan from the Amazon Web Services Support Center . For more information on Amazon Web Services Support plans, see Compare Amazon Web Services Support plans .

3. For more information on Monitoring Amazon Web Services Trusted Advisor check results with Amazon EventBridge, review the Amazon Web Services Trusted Advisor and EventBridge page.

A more detailed step by step guide to deploy and test the solution using Amazon Web Services CloudFormation is available in Amazon Web Services Trusted Advisor GitHub .

Deploying the solution in a multi-account Amazon Web Services Organization

Amazon Web Services Organizations helps you centrally govern your environment as you grow and scale your workloads on Amazon Web Services.

To implement the solution across multiple accounts within Amazon Web Services Organizations, it is necessary to consolidate Amazon Web Services Trusted Advisor events from each individual account into a designated central or governance account.

The following diagram (Figure 2) illustrates the solution architecture to extend a solution to Amazon Web Services organization with multiple accounts. Amazon Web Services Trusted Advisor events are sent to the “default” event bus of each member account. The events are forwarded to a target event bus in EventBridge of the central governance account. An EventBridge rule in the central governance account triggers Amazon Web Services Step Functions to initiate the remediation workflow in response to events received from participating member accounts.

Figure 2: Architecture for deploying across multiple Organization accounts

A step by step approach to centralize EventBridge events from member Amazon Web Services Organizations accounts into a central account using Amazon Web Services CloudFormation StackSets and CDK is discussed here .

Customizing the solution

We have discussed an approach to automatically remediate the Exposed Access keys recommendation from Trusted Advisor’s security pillar. This architecture pattern can be extended to automatically remediate other security recommendations, and recommendations from Trusted Advisor’s fault tolerance, cost optimization, and performance pillars. To do so, identify the desired Trusted Advisor recommendation to auto-remediate. The recommendation name can be found in the “ check-name ” field of the event delivered by EventBridge. See the sample event under the “Solution Workflow” above for reference. Implement custom remediation actions in the associated target Lambda function by following the approach as discussed in the sections above.

For other examples to auto-remediate deviations detected by Amazon Web Services Trusted Advisor recommendations visit Amazon Web Services Trusted Advisor Tools GitHub .

Conclusion

In this blog post, we detailed a solution to automate remediation of Amazon Web Services Trusted Advisor recommendations. The solution is applied to a single Amazon Web Services account and Amazon Web Services Organization. The solution is useful for Amazon Web Services Trusted Advisor recommendations that need rapid response times and continuous protection of your Amazon Web Services resources. It eliminates manual processes as they can be prone to human error. Automatic remediation also allows for scalability by enabling simultaneous resolution across multiple resources or accounts, and reducing the burden on administrators. This proactive approach reduces the window of vulnerability, minimizes deviations from Amazon Web Services best practice recommendations, and maintains a secure and optimized environment.

About the authors: