VPC endpoint considerations for upgrading or creating Amazon Web Services DMS version 3.4.7 or higher

by Sushant Deshmukh , Aritra Biswas , and Arnab Saha | on

Amazon Web Services Database Migration Service (Amazon Web Services DMS) version 3.4.7 and above includes support for Amazon Virtual Private Cloud (Amazon VPC) VPC endpoints and makes it easier to maintain end-to-end network configuration within the same Amazon Web Services region for replication tasks. To learn more about VPC endpoints, refer to Connect your VPC to services using Amazon Web Services PrivateLink .

In this post, we demonstrate an Amazon Web Services DMS task failure caused by a network issue that you may encounter while upgrading your Amazon Web Services DMS replication instance to version 3.4.7 or higher. We also show how to mitigate this issue by configuring a VPC endpoint and upgrading the Amazon Web Services DMS replication instance for seamless replication.

Solution overview

VPC endpoints for all source and target endpoints ensures that all your traffic remains within your VPC and under your control. It reduces replication interruptions and improves the quality of the data transfer. With VPC endpoints, your Amazon Web Services DMS replication instance doesn’t require a public IP address to communicate with Amazon Web Services services such as Amazon Simple Storage Service (Amazon S3), Amazon Kinesis , Amazon Web Services Secrets Manager , Amazon DynamoDB , Amazon Redshift , and Amazon OpenSearch Service .

The following diagram illustrates the solution architecture.

Prerequisites

To follow along with this post, you should have the following prerequisites:

  • A private Amazon Web Services DMS replication instance with version 3.4.6 (Public accessibility turned off)
  • Any Amazon Web Services DMS supported source (For the purposes of the post, we use an Amazon Relational Database Service (Amazon RDS) for MySQL database as a source database)
  • An Amazon S3 bucket as a target for Amazon Web Services DMS task
  • An Amazon Web Services DMS replication task
  • An Amazon Web Services Identity and Access management (IAM) role called dms-vpc-role with policy AmazonDMSVPCManagementRole
  • A VPC endpoint

Migration with Amazon Web Services DMS 3.4.6

We created an Amazon Web Services DMS task for a sample (DMS_SAMPLE) schema and performed a full load and change data capture (CDC) to migrate data from the source database to the Amazon S3 target.

From the Amazon Web Services DMS task logs (see the following screenshot), we can see that the task for table TESTING_4 is running fine.

Next, we upgrade the Amazon Web Services DMS replication instance to 3.4.7.

Upgrade the Amazon Web Services DMS replication instance to 3.4.7

You can upgrade the replication instance by modifying the replication instance and specifying the replication engine version as 3.4.7.

After the upgrade is complete and the task is resumed, you see the following error messages in the Amazon Web Services DMS task logs.

The reason for the failure is that, for private Amazon Web Services DMS Replication instances prior to Amazon Web Services DMS version 3.4.7, the replication network traffic for the S3 endpoints were routed through the Amazon backbone network. Starting with Amazon Web Services DMS version 3.4.7 and above, you need VPC endpoints or an internet gateway route in the route tables used by Amazon Web Services DMS replication instance to access the Amazon S3 target. VPC endpoints powered by Amazon Web Services PrivateLink allow you to connect directly and privately to Amazon Web Services services like Amazon S3 from within your VPC.

In the next step, we create a VPC endpoint.

Create a VPC endpoint

To create a VPC endpoint, complete the following steps:

  1. On the Amazon VPC console, choose the same Region as your Amazon Web Services DMS replication instance.
  2. In the navigation pane, choose Endpoints .
  3. Choose Create endpoint .
  4. For Service category , select Amazon Web Services services .
  5. Under Services , filter for Amazon S3.
  6. Select the Amazon S3 internet gateway.

This creates a gateway endpoint that sends traffic to Amazon S3 using a private IP addresses. You route traffic from your VPC to the gateway endpoint using route tables which we select below.

  1. For VPC , choose the same VPC as the Amazon Web Services DMS replication instance.
  2. For Route tables , select the applicable routes.
  3. Under Policy , select Full access.
  4. After the endpoint is created, verify the status shows as Available.

Test the Amazon S3 connection endpoint

Test the Amazon S3 target endpoint with the newly upgraded 3.4.7 replication instance and check if it’s successful.

Now you can resume/restart the Amazon Web Services DMS task and verify the task is running fine. You can also, check the Amazon Web Services DMS task CloudWatch logs to confirm that the task is running as expected.

Clean up

Remove the resources that were created when testing the upgrade to avoid any ongoing charges.

Recommendations when upgrading an Amazon Web Services DMS Replication Instance from 3.4.6 to 3.4.7

  1. Create a VPC Endpoint
  2. Stop the Amazon Web Services DMS task/s that are currently running on the Amazon Web Services DMS replication instance you wish to upgrade
  3. Upgrade the Amazon Web Services DMS Replication Instance from 3.4.6 to 3.4.7
  4. Resume the Amazon Web Services DMS task/s that were stopped

Summary

In this post, we provided step-by-step guidance on safely upgrading your Amazon Web Services DMS replication instance to 3.4.7 and using VPC endpoints to communicate with Amazon S3 as a target endpoint.

Leave a comment if you have questions or suggestions.

About the authors

Sushant Deshmukh is a Database Consultant with Amazon Web Services Professional Services. Sushant works with Amazon Web Services customers and partners to build highly available, scalable and secured database architectures on Amazon Web Services. Also, helping customers migrate and modernize their databases to Amazon Web Services Cloud. Outside of work, he enjoys traveling, exploring new places, playing volleyball and spending time with his family and friends.

Aritra Biswas is a Cloud Support DBA with Amazon Web Services and Subject Matter Expert for Amazon Web Services Database Migration Service, he has over a decade of experience in working with relational databases. At Amazon Web Services, he works with Service Teams, Technical Account Managers, Solutions Architects, and assists customers migrate database workloads to Amazon Web Services. Outside of work he enjoys playing racquetball and spending time with family and friends.

Arnab Saha is a Senior Database Specialist Solutions Architect at Amazon Web Services. Arnab specializes in Amazon RDS, Amazon Aurora , Amazon Web Services DMS and Amazon Elastic Block Store. He provides guidance and technical assistance to customers thus enabling them to build scalable, highly available and secure solutions in Amazon Web Services Cloud.

The mentioned AWS GenAI Services service names relating to generative AI are only available or previewed in the Global Regions. Amazon Web Services China promotes AWS GenAI Services relating to generative AI solely for China-to-global business purposes and/or advanced technology introduction.