Inspired by Steph Gooch’s 2022 re:Invent session “Optimize the others” chalk talk session
If you can’t measure it, you can’t manage it. In order to optimize your cloud resources efficiency, you need to be able to view and understand cloud cost and usage data. Often times, you focus on the top spenders, because they drive up your bills and are easy to identify. For example, when filtering your Amazon Web Services cost by service in
Amazon Web Services Cost Explorer
, you usually see the cost breakdown by services, such as,
Amazon EC2
-instances,
Amazon Simple Storage Service
(S3), and
Amazon Relational Database Service
(RDS). The rest of the services are lumped together in a category called “Others” in the graph. However, the cost of the “Others” category can be just as high as the top cost drivers. It’s worth looking into the sources of these costs and identifying opportunities for cost and performance optimization. In this blog, I’ll use a few examples to demonstrate how you can dive deeper and understand the cost elements of these “other” services and what you can do to optimize the spend.
Speaking of the services that usually fall under the category of “Others”, I want to start with services that are labeled as “ EC2-Other ”. “EC2-Other” tracks costs associated with the
Amazon EC2
instances, such as,
Amazon EBS volumes
and
snapshots
,
elastic IP address
,
data transfer
,
NAT gateways
, and more. You can view the details of your “EC2-Other” spend by grouping with “usage type” in Cost Explorer. Services, such as,
Amazon Web Services Compute Optimizer
and
Amazon Web Services Trusted Advisor
, offer
recommendations for EBS volume usage
, including detecting unattached EBS volumes, and migrating from one volume type to another, e.g. General Purpose SSD (gp2) to gp3. EBS snapshots back up data from your EBS volumes and are charged as incremental data changes made to your EBS volumes. To save costs on EBS snapshots, you can start with single snapshots that were created with your
AMI (Amazon Machine Image)
. When creating an AMI for an existing EC2 instance, Amazon EC2 creates EBS snapshots of your instances’ root volume and other EBS volumes attached to your instance. However, when you deregister your AMI, but did not opt to delete the snapshot, that snapshot will persist and cause full storage charge. Make sure you deregister the AMI and delete the associated snapshots, when you no longer need them. You can automatically create, retain, and delete EBS snapshots and EBS-backed AMIs with
Amazon Data Lifecycle Manager
.
Now, let’s talk about services that are categorized under “Others”. Services, such as
Amazon CloudWatch
,
Amazon Web Services CloudTrail
, and
Amazon Web Services Lambda
, are usually on the list. CloudWatch monitors the performance of your Amazon Web Services resources and applications run on Amazon Web Services. CloudWatch metrics enable many other Amazon Web Services services, such as Amazon Web Services Compute Optimizer, to provide optimization recommendations. However, CloudWatch itself incurs costs, if your usage exceeds the
free tier limit
. For instance, many Amazon Web Services services send basic monitoring metrics to CloudWatch at no charge. The basic monitoring is enabled by default once you start using these services. But if you choose to enable detailed monitoring for services that offer the option, for example, sending EC2 metrics more frequently at one-minute intervals, instead of the five-minute intervals used in the basic monitoring, you’ll incur EC2 detailed monitoring cost (typically charged at $2.10 per instance per month and goes down to $0.14 per instance at the lowest priced tier). If you no longer require detailed monitoring metrics for your EC2 instances, you should turn off the detailed monitoring. If you’ve set up a
CloudWatch alarm
that is associated with multiple metrics, you should be aware that each alarm-metric incurs cost. Furthermore,
CloudWatch logs
, custom and vended logs, are other resource usage type that can incur ingestion charges based on the volume of the ingested/scanned log data, or archival charges, if you forget to delete data after certain retention period (the default retention policy on log groups is set to “Never Expire”). Actively manage how your CloudWatch detailed metrics, alarms, and logs are configured, based on your applications’ needs, so you can reduce your CloudWatch costs. Learn ways you can reduce your CloudWatch charges from this knowledge center article “
How can I determine why I was charged for CloudWatch usage, and then how can I reduce future charges
?”
As CloudWatch tracks the performance of Amazon Web Services services and resources in your environment, CloudTrail monitors the API activities made in your Amazon Web Services environment for governance and auditing purposes. The service is free to use, when there is only a single copy of management events delivered via trail to your S3 bucket. However, several use cases will incur cost, for instance, if you create additional copies of management events at the organizational or account level. CloudTrail also incurs costs when you record and analyze events using
CloudTrail Lake
, or choose to analyze events using
CloudTrail Insights
. Due to the fact CloudTrail is enabled by default, it is possible that additional copies are created at the organizational and account level. One way to identify any duplicated management event records is to look for the “PaidEventsRecorded” metric under your CloudTrail service charges on your Amazon Web Services bill to see if there are additional copies of management events recorded in any specific regions (Image 1). For intentional control of what management and data events to log, you can use CloudTrail
Event Selector
and
Advanced Event Selector
to specify the logging criteria. Work with your Cloud Center of Excellence Office to centrally manage CloudTrail and identify who owns duplicated CloudTrails. If you decide to remove additional trails, make sure you delete the S3 buckets and objects attached to it, so you are not continuously paying for logs that you no longer need.
Image 1. CloudTrail service charge on Amazon Web Services Bill
Many of you are aware of the cost saving advantage by leveraging serverless applications and the gained operational efficiency for the ongoing maintenance and infrastructure. Amazon Web Services Lambda , the serverless, event-driven, compute service, allows you to run code at scale without provisioning or managing infrastructure. Yet, it’s helpful to understand the key components of its pricing structure, for example, the number of requests, duration of invocations, and the configured memory allocation. You can optimize the memory configuration of Lambda functions with Amazon Web Services Compute Optimizer or Amazon Web Services Lambda Power Tuning. To increase the efficiency of the invocation, besides applying
best practices
to your Lambda code, you can set realistic function timeout to prevent prolonged invocation. Choose Graviton2 architecture for arm64-based infrastructure and gain up to 19 percent better performance at 20 percent lower cost. For steady Lambda workloads, consider Compute Savings Plans for discounted rates of up to 17% for a 1 or 3 year term, and enjoy the flexibility to apply the benefits across EC2 and Fargate. Understand what its pricing is comprised of and leverage services, such as, Compute Optimizer, for usage optimization, and purchase option (Savings Plans) for discounts. Learn about other techniques to reduce your Amazon Web Services Lambda costs from this
blog post
.
Conclusions
In the blog post, I used a few examples to show how you can zoom into the services that typically fall under the “others” category and understand their cost elements and ways to drive down the cost. It’s helpful to have dedicated FinOps professionals to frequently audit the usage efficiency and work across teams to implement necessary changes. Hope the blog post can help you get started and discover cost saving opportunities that fit your application and business needs.
