Estimating Amazon Web Services Config recorder costs and usage using Amazon Web Services CloudTrail

Amazon Web Services Config is a service that tracks configuration changes of Amazon Web Services resources in your Amazon Web Services account.  Amazon Web Services Config uses the configuration recorder to create a configuration item whenever it detects a change to a resource type that it is recording. For example, if Amazon Web Services Config is recording Amazon S3 buckets, Amazon Web Services Config creates a configuration item whenever a bucket is created, updated, or deleted. A part of Amazon Web Services Config pricing is based on the number of configuration items recorded in your account.  However, configuration items are dependent on workload behavior which can make it difficult to estimate the number of configuration items.

In this post, we will show how you can use Amazon Web Services CloudTrail to help get an estimate of the number of configuration items if Amazon Web Services Config is enabled.  Amazon Web Services CloudTrail is a service that monitors and records account activity across your Amazon Web Services infrastructure. These recorded events will help us determine an estimate of configuration items that could get recorded by Amazon Web Services Config.  If you are utilizing Amazon Web Services Config rules or Amazon Web Services Config conformance packs, there will be additional charges. Please see the Amazon Web Services Config pricing page for examples.

Step 1: Querying Amazon Web Services CloudTrail

Prerequisites

The sample CloudTrail queries in this blog require customers have either of the following setup options:

1. CloudTrail Lake  event data store setup to receive CloudTrail Events.
2. Amazon Athena setup to query your CloudTrail logs .

The option you choose will depend upon which of these options you have configured in your environment

Option 1: Running the query in Amazon Web Services CloudTrail Lake

The following sample CloudTrail Lake query will return the total captured API events related to a list of Amazon Web Services services within a given time period.  You can optionally add or remove additional Amazon Web Services services to the query by updating the eventSource section of the query. The API events queried are events that could trigger a resource change in Amazon Web Services Config.  There are some resource types that are not captured by the sample query that Amazon Web Services Config records such as the Amazon Web Services::Config::ResourceCompliance. This query will give you a starting point as to how many configuration items could get recorded if the Amazon Web Services Config recorder was turned on.  The Amazon Web Services Calculator can help you create an estimate for Amazon Web Services Config. Please note running this query will incur an additional cost, please review the CloudTrail pricing page for the updated pricing.

1. Navigate to the CloudTrail console .
2. In the left-hand navigation menu, choose Lake.
3. Choose the Editor
4. Copy the below query and paste it into your editor window. (Note: you must replace $EDS_ID with the id of your event data store. Also, you can add additional eventSource you would like to track)

SELECT
    recipientAccountId, awsRegion, eventSource, count(* 
    ) as TotalPossibleCI 
FROM
    $EDS_ID
Where
    (eventSource like 'eks%' 
        or eventSource like 'ec2%' 
        or eventSource like 'vpc%'
        or eventSource like 'ecs%' 
        or eventSource like 'iam%' 
        or eventSource like 'autoscaling%' 
        or eventSource like 's3%' 
        or eventSource like 'rds%' 
        or eventSource like 'backup%' 
        or eventSource like 'athena%' 
        or eventSource like 'cloudtrail%' 
        or eventSource like 'cloudfront%' 
        or eventSource like 'cloudformation%' 
        or eventSource like 'code%' 
        or eventSource like 'ecr%' 
        or eventSource like 'lambda%' 
        or eventSource like 'efs%' 
    ) 
    and readOnly=False 
    and managementEvent=True 
    and eventTime > '2023-04-01 00:00:00' 
    AND eventTime < '2023-04-30 00:00:00' 
group
    by recipientAccountId, awsRegion, eventSource Order by recipientAccountId desc, TotalPossibleCI desc

5. Next, you must replace the time range that will be searched with the time range you want to use. This will help to reduce the amount of data scanned within the query to only search the time and date specified. The date string specified after eventTime > is the earliest event timestamp that will be included, while the date string specified after eventTime < is the latest event timestamp that will be included.
6. Click Run and then your results will show under Query Results. Please note the value of the TotalPossibleCI column will be used when estimating the Amazon Web Services Config cost for the total amount of configuration items recorded within a month.

Figure 1: Amazon Web Services CloudTrail Lake query results

7. You can then save the query run again at future time, by clicking the Save
8. Give it the name Estimate Amount of Amazon Web Services Config Configuration Items.
9. Enter a description of Estimate of Amount of Amazon Web Services Config Configuration Items.
10. Click Save query.

Option 2: Amazon Web Services CloudTrail query for Amazon Athena

To run a similar query if you have trails for CloudTrail setup to use Amazon Athena .  You can use the below sample Athena query to give you an estimate of how many potential configuration items could be created if Amazon Web Services Config would be enabled.  Similar to the CloudTrail Lake query above, you can optionally add or remove additional Amazon Web Services services to the query by updating the eventSource section of the query. The results of the query will give you a starting point that can be used with the Amazon Web Services Calculator to estimate Amazon Web Services Config cost for the amount of configuration items that could get recorded if the Amazon Web Services Config recorder was turned on.  Also, please note that running this query will incur an additional cost, please review the Athena pricing page for the updated pricing.

(Note: you must replace ${tableName} with the Athena table name for CloudTrail and adjust the eventTime fields with the date range you would like to query. Also, you can add additional eventSource you would like to track)

SELECT
  recipientAccountId, awsRegion, eventSource, count(*) as TotalPossibleCI
FROM
  ${tableName}
WHERE
  (eventSource like ‘eks%’
    or eventSource like ‘ec2%’
    or eventSource like ‘vpc%’
    or eventSource like ‘ecs%’
    or eventSource like ‘iam%’
    or eventSource like ‘autoscaling%’
   or eventSource like ‘s3%’
   or eventSource like ‘rds%’
   or eventSource like ‘backup%’
   or eventSource like ‘athena%’
   or eventSource like ‘cloudtrail%’
   or eventSource like ‘cloudfront%’
   or eventSource like ‘cloudformation%’
   or eventSource like ‘code%’
   or eventSource like ‘ecr%’
   or eventSource like ‘lambda%’
   or eventSource like ‘efs%’
  )
  AND readonly = ‘false’
  AND managementEvent = ‘True’
  AND eventTime > ‘2023-04-01 00:00:00’
  AND eventTime < ‘2023-04-30 00:00:00’
GROUP BY recipientAccountId, awsRegion, eventSource
ORDER BY recipientAccountId desc, TotalPossibleCI desc;

Step 2: Create Estimate with Amazon Web Services Pricing Calculator

We can use Amazon Web Services Pricing Calculator to create an estimate of the monthly Amazon Web Services Config cost for the amount of possible configuration items recorded if the Amazon Web Services Config recorder was turned on.  Please note, that configurations items recorded are just one part of the Amazon Web Services Config pricing, you would also need to estimate the amount of Amazon Web Services Config rules and conformance pack evaluation that will be ran in your account.

1. Navigate to Amazon Web Services Pricing Calculator page and choose Create an Estimate.
2. In the Find Service search box under Select Service, search for “Amazon Web Services Config” and under Amazon Web Services Config, choose Configure.
3. In the Configure Amazon Web Services Config section, type a description and choose the region you would like to create the pricing estimate for.
4. In the Configuration items recorded section, under the Number of Configuration items recorded, enter in the total value of “TotalPossibleCI” from the query you ran earlier in CloudTrail.
5. This will then give you an estimate of the Amazon Web Services Config cost for the total amount of configuration items recorded within a month.

Figure 2: Creating an Estimate with Amazon Web Services Pricing Calculator

Cleanup

If you like to remove the query saved in CloudTrail Lake, you can do the following steps:

1. Navigate to the CloudTrail console .
2. In the left-hand navigation menu, choose Lake.
3. Choose the Saved queries
4. Select the Estimate Amount of Amazon Web Services Config Configuration Items
5. Click Delete.

Conclusion

This post demonstrates how to use CloudTrail Lake to help gather statistic related to the amount of API related events that could trigger a creation of a configuration items by Amazon Web Services Config.  With CloudTrail Lake you can further use additional queries to gather deeper insight to activity being captured within your account. To learn more about CloudTrail Lake see the blog article Announcing Amazon Web Services CloudTrail Lake – a managed audit and security Lake .

About the authors: