Accelerate database migration planning with Amazon Web Services DMS Fleet Advisor

by Amit Arora and Shashank Kalki | on

Planning a migration journey with extensive on-premises database and analytics resources have many requirements such as including time to generate a migration plan and expertise in compiling inventory, designing a migration strategy, and finalizing migration targets. Transition to Amazon Web Services became remarkably streamlined with Amazon Web Services DMS Fleet Advisor service. Amazon Web Services DMS Fleet Advisor helps you construct a database inventory, performing in-depth schema analysis, and recommending right-sized target databases. It reduces migration efforts from weeks or even months to hours. Amazon Web Services DMS Fleet Advisor collects performance metrics and usage patterns, and combines this information with database license and feature information to recommend potential database engine and instance options for migration to Amazon Web Services.

In this post, we cover features of Amazon Web Services DMS Fleet Advisor, how to configure it, and how to generate an inventory of databases and analyze the data to provide target recommendations in Amazon Web Services.

Solution overview

Before we take a deep dive into configuration settings and report generation, let’s first understand the architecture of Amazon Web Services DMS Fleet Advisor. Amazon Web Services DMS Fleet advisor is composed of a data collector, Amazon Simple Storage Service (Amazon S3) bucket and Amazon CloudWatch . A data collector installed on the source server collects various metrics and stores them in an Amazon S3 bucket. This data is analyzed by Amazon Web Services DMS Fleet Advisor and presented as recommendations. The following diagram shows how Amazon Web Services DMS Fleet Advisor works.

Amazon Web Services DMS Fleet Advisor’s data collector supports Lightweight Directory Access Protocol (LDAP) for discovery. It gathers various metrics from the operating system and database servers to recommend potential database engine and instance options for migration to Amazon Web Services. It also highlights migration limitations based on database features in use and provides the monthly estimated cost of instances in Amazon Web Services. Amazon Web Services DMS Fleet Advisor collects various metrics to build inventory, analyze schemas and generate target recommendations.

Configure Amazon Web Services DMS Fleet Advisor

We walk you through the following steps to set up and use Amazon Web Services DMS Fleet Advisor:

  1. Configure a DMS data collector to discover OS and database servers.
  2. Discover servers.
  3. Collect metadata and metrics.
  4. Analyze the output.
  5. Generate target recommendations.

Set up Amazon Web Services resources for Amazon Web Services DMS Fleet Advisor

Before we run Amazon Web Services DMS Fleet Advisor, you must configure the following Amazon Web Services resources:

  • Amazon S3 bucket
  • Amazon Web Services Identity and Access Management (IAM) policies, user, and role
  • Database users
  • Data collector permissions
  • Amazon CloudWatch metrics

Create an S3 bucket

To create an S3 bucket where inventory metadata can be stored, complete the following steps:

  1. On the Amazon S3 console, choose Buckets in the navigation pane.
  2. Choose Create bucket .
  3. For Bucket name , enter a globally unique name, such as <YOUR BUCKET NAME> .
  4. For Amazon Web Services Region , choose the Region where you use Amazon Web Services DMS Fleet Advisor.
  5. Keep the remaining settings as default and choose Create bucket .

Create IAM resources

To make sure that your data collector works correctly and uploads the collected metadata to your S3 bucket, you must create IAM policies, a user, and a role. Complete the following steps:

  1. Create two IAM policies :
    • The first policy allows Amazon Web Services DMS Fleet Advisor and the data collector to access the S3 bucket.
    • The second policy allows the DMS data collector to access Amazon Web Services DMS Fleet Advisor.
  2. Create a user with minimum permissions to use the DMS data collector and attach the two policies created in previous step.

The IAM role you create encapsulates the policies and allows Amazon Web Services DMS Fleet Advisor and your data collector to access Amazon S3.

  1. Create your IAM role and attach the policies you created.

Create database users

Create users for your source databases MySQL , Oracle , PostgreSQL , SQL Server with the required permissions for the DMS data collector. Upon completion of data collection step or analysis steps, you can delete the database users created for the DMS data collector for all source databases .

Configure DMS data collector permissions

To perform the discovery process efficiently, the DMS data collector requires additional permissions .

Configure CloudWatch metrics

Amazon Web Services DMS Fleet Advisor uses a service-linked role to streamline the management of CloudWatch metrics in your Amazon Web Services account. With this role, Amazon Web Services DMS Fleet Advisor publishes the database performance metrics it collects to CloudWatch on your behalf. After you establish the service-linked role, you have instant access to performance metrics for your source databases in target recommendations. Additionally, you can monitor and analyze these metrics within your CloudWatch, ensuring comprehensive visibility and control.

Configure a DMS data collector

A data collector is a Windows application that you install in your local environment. This application connects to your data environment and collects metadata and performance metrics from your on-premises database and analytic servers. Then Amazon Web Services DMS Fleet Advisor utilizes the collected metrics to build an inventory of servers, databases, and schemas that you can migrate to the Amazon Web Services Cloud.

Create and download a data collector

To set up your data collector, complete the following steps:

  1. On the Amazon Web Services DMS console , choose Data collectors under Discover in the navigation pane.
  2. Choose Create data collector .
  3. For Name , enter a name for your data collector.
  4. For Amazon S3 bucket , enter your bucket details. Make sure your bucket is in the same Region where Amazon Web Services DMS Fleet Advisor is running.
  5. For IAM role , choose the IAM role you created as a prerequisite.
  6. Choose Create data collector .

The Data collectors page opens and the new data collector appears on the list.

  1. Choose Download local collector on the information banner to download your newly created data collector.

A message informs you that the download is in progress. When the download is complete, you can access the AWS_DMS_Collector_Installer_version_number.msi file.

Install and configure the data collector

To install a data collector on a client in your network, run the .MSI file you downloaded the previous step. After you install the data collector, you can run it from a browser by entering http://localhost:11000/ as the address. The initial DMS data collector run needs you to configure credentials to sign in to the data collector. Configure data forwarding by providing IAM access key and secret key (created in the prerequisites) and choose the data collector. For more details, refer to Installing and configuring a data collector .

Verify that access to Amazon S3 and access to Amazon DMS is set to Yes and save the configuration.

Discover servers

You can use the DMS data collector to find and list all available servers in your network. To discover OS servers, you need permission to run remote PowerShell, SSH, and WMI scripts and commands, as well as access to the Windows registry.

To get started with the DMS data collector, complete the following tasks:

  1. On the DMS data collector, choose Discovery in the navigation pane.
  2. On the OS servers tab, enter the details of the LDAP server and choose Run discovery . You don’t need to start from an LDAP server, and you can instead import the list of OS servers to perform database discovery.

  1. On the OS servers tab, select each server that you want to monitor and add to the monitored object.
  2. For the monitored objects, verify the connection status by choosing Verify Connection on the Actions menu for each server in the list and view the results in the Connection status column.

If you get any issues in the connection status, edit the credentials and fix these issues before proceeding to the next step.

  1. After the monitored objects connection is verified, choose Run discovery .
  2. Select the database servers for data collection

  1. Verify the connections to the monitored databases.

Collect data

To start collecting data, select the objects on the Monitored objects page and choose Run data collection . The DMS data collector can collect from up to 100 databases at one time. The data collector can work in two modes: single run or ongoing monitoring. Choose one of the following modes for data collection:

Metadata and database capacity (default option) – The DMS data collector collects information from the database or OS servers. It includes schemas, versions, editions, CPU, memory, and disk capacity. You can compute target recommendations in Amazon Web Services DMS Fleet Advisor based on this information. If the source database is over-provisioned or under-provisioned, then the target recommendations also will be over-provisioned or under-provisioned.

Metadata, database capacity, and resource utilization – In addition to metadata and database capacity information, the DMS data collector collects actual utilization metrics of CPU, memory, IOPS, I/O throughput, disk storage, active database server connections for the databases or OS servers. The target recommendations provided will be more accurate because they are based on the actual database workloads.

For any issues while collecting data, refer troubleshooting for DMS data collector .

Amazon Web Services DMS Fleet Advisor gathers the above-mentioned metrics during ongoing monitoring. You can run your DMS data collector for a period of 1–60 days. Though general recommendation for data collection is at least 7 days but we recommend you select a data collection period that accurately captures the database workloads for your specific case. For example, if you have peak database workloads for a few days at the end of the month, then ensure data collector is on for those few days.

Analyze the output

Amazon Web Services DMS Fleet Advisor analyses database schemas and shares information to understand which are the good candidates for migration. The analysis examines the resemblance between various schemas by comparing the names of schema objects, such as functions and procedures but not actual code of the objects. This process helps identify similar objects present in different database schemas. You can export inventory analysis information into a .csv file for deep dive.

Database inventory

To access a list of databases obtained from your network servers, navigate to the Inventory page and choose the Databases tab. The database inventory displays information such as schema count, engine name, version, edition, and data collector name. If a particular engine version is on extended support or has reached its end of life (EOL), it will show in red.

Schema inventory

To access a list of schemas obtained from your network servers, navigate to the Inventory page and choose the Schemas tab. The schema inventory displays information such as server name, database name, engine, complexity, and similarity in comparison to the original schema. Similarity comparison between the schemas across all the databases explains the amount of efforts required to perform the migration of all similar database storage & code objects in the schema.

Target recommendations

Amazon Web Services DMS Fleet Advisor provides recommendations for the databases discovered, suggesting one or more potential Amazon Web Services target engines. Among the multiple recommended target engines, Amazon Web Services DMS Fleet Advisor selects a single target engine that is deemed an appropriate migration destination in terms of size. For instance, when dealing with PostgreSQL and MySQL source databases, Amazon Web Services DMS Fleet Advisor may suggest including Amazon Aurora and Amazon Relational Database Service (Amazon RDS) DB instances as target options, and highlight one of them as the recommended choice. It provides estimated monthly cost of the target instances based on the region where Amazon Web Services DMS Fleet advisor is hosted. To further optimize the estimated cost of the recommendations you can export the estimate to Amazon Web Services Pricing Calculator .

The recommendations generated by Amazon Web Services DMS Fleet Advisor are derived from either the capacity or resource utilization of the databases. If you opt for recommendations based on database capacity, Amazon Web Services DMS Fleet Advisor matches gathered total capacity metrics of the source database to the most suitable instance class. On the other hand, if you choose recommendations based on resource utilization, the Amazon Web Services DMS Fleet Advisor calculates the 95th percentile value for utilization metrics such as CPU, memory, disk storage, and IOPS, and then matches it to the closest instance class. You can check the status and type of collected metrics at Generate recommendations and Recommendation detail in Amazon Web Services DMS Fleet Advisor as well as in your Amazon CloudWatch .

Refer Amazon Web Services DMS Fleet advisor target recommendation limitations for any issues.

Clean up

Upon completion of data collection and analysis on Amazon Web Services DMS Fleet advisor target recommendations, it is time to perform a cleanup and free up resource.

  1. Stop DMS data collector installed in your local environment.
  2. In the Amazon Web Services DMS navigation pane, click on Data Collectors . Select the data collector you plan to clean up and delete.

  1. Finally, perform a clean-up on the Amazon S3 bucket where the data collector files are stored. In the Amazon S3 bucket console, select the intended S3 bucket and select delete.

Summary

In this post, we have laid out the step by step process to set up and customize Amazon Web Services DMS Fleet Advisor . We have covered preliminary steps, installation and configuration an agent, and seamless integration with Amazon Web Services DMS console. We have also walked you through running Amazon Web Services DMS Fleet Advisor console to analyze the data collected and generate target recommendations. The Amazon Web Services DMS Fleet Advisor team consistently enhances the product, introducing new features in upcoming quarters and offering additional support for various migration scenarios. Check out Amazon Web Services DMS Fleet Advisor updates with each release for the latest improvements .

About the Authors

Amit Arora is a Solutions Architect with a focus on database and analytics at Amazon Web Services. He works with our financial technology and global energy customers and Amazon Web Services certified partners to provide technical assistance and design customer solutions on cloud migration projects, helping customers migrate and modernize their existing databases to the Amazon Web Services Cloud.

Venkata Shashank Kalki is a Solutions Architect with the Amazon Database Migration Accelerator team at Amazon Web Services. He works as an Amazon DMA Advisor to help Amazon Web Services customers migrate their on-premises data to Amazon Web Services Cloud database solutions.

The mentioned AWS GenAI Services service names relating to generative AI are only available or previewed in the Global Regions. Amazon Web Services China promotes AWS GenAI Services relating to generative AI solely for China-to-global business purposes and/or advanced technology introduction.