Introducing private DNS support for Amazon S3 with Amazon Web Services PrivateLink

by Sumiran Tandon, Rohit Aswani, and Shubham Singh | on

Compliance requirements often mandate private connectivity when on-premises applications use cloud storage. To satisfy these requirements, customers set up private connections to Amazon S3 using Amazon Web Services PrivateLink over either Amazon Web Services Direct Connect , or Amazon Web Services Site-to-Site VPN . As a result, data is transmitted directly to and from Amazon Web Services, never traversing the public internet. Amazon Web Services PrivateLink lets you provision interface endpoints in your Amazon Virtual Private Cloud (Amazon VPC) , which assign private IP addresses to S3. PrivateLink automatically provisions globally unique public DNS names for these private IPs, which your applications can use to access S3. While you can create custom DNS entries on-premises to point on-premises clients at these private IP addresses when using S3’s regional names (s3.<Region>.amazonaws.com), this adds operational overhead and is difficult to manage, and as a result, this is not a preferred option.

To help simplify your DNS configuration for private connectivity, Amazon S3 now supports private DNS options for your S3 interface VPC endpoints . With private DNS for S3, your on-premises applications can use Amazon Web Services PrivateLink to access S3 over an interface VPC endpoint, while requests from your in-VPC applications access S3 using gateway VPC endpoints . Routing requests like this helps you take advantage of the lowest-cost private network path without having to make code or configuration changes to your clients.

In this blog post, we demonstrate how to take advantage of private DNS to access Amazon S3 using Amazon Web Services PrivateLink. We also discuss configuration options for various scenarios, and how to verify that your clients are connecting to Amazon S3 over gateway VPC endpoints and interface VPC endpoints.

VPC endpoints for Amazon S3

There are two types of VPC Endpoints that you can use to connect to Amazon S3 from your VPC: gateway endpoints and interface endpoints.

Gateway VPC endpoints provide reliable connectivity to Amazon S3 and DynamoDB without requiring an internet gateway or a NAT device for your VPC. Gateway VPC endpoints can be set up with a few clicks in the Amazon Web Services Management Console, and use your VPC route table to route requests from clients within your VPC to S3’s or DynamoDB’s public IPs, over the Amazon Web Services network.  Gateway VPC endpoints have no additional charge and support connectivity only from resources local to the respective VPC where the gateway endpoint is created.

Interface VPC endpoints provide private connectivity for over 140 Amazon Web Services services and third-party SaaS applications using Amazon Web Services PrivateLink. An interface VPC endpoint creates an elastic network interface (ENI) with a private IP address in your VPC subnets. Interface VPC endpoints support connectivity from on-premises over Amazon Web Services Direct Connect or Amazon Web Services Site-to-Site VPN. When you configure an interface VPC endpoint, we create PrivateLink provided public DNS names for your endpoint that are resolvable both within the VPC and from on-premises. Interface VPC endpoints have two charges – an hourly charge for each VPC endpoint provisioned in each Availability Zone, and a per-GB data processing charge. For pricing details, please visit Amazon Web Services PrivateLink pricing .

The most cost-efficient way to access S3 is to use Gateway VPC endpoints where possible (e.g., from EC2 instances in the Region) and to use Interface VPC endpoints from other locations such as from on premises.

Accessing Amazon S3 interface endpoint with private DNS name

Figure 1 shows a hybrid network setup where you connect from on-premises over an Amazon Web Services Direct Connect connection or Amazon Web Services Site-to-Site VPN connection. In this setup, you configure Amazon Route 53 inbound Resolver endpoints and set up conditional forwarding on your on-premise DNS resolver to forward DNS queries to the private IP addresses of inbound resolver endpoints. Then, when you create an interface VPC endpoint for S3, you will have an option to enable a private DNS name.

Figure 1 - Setup when connecting from on-premises over Direct Connect or Site-to-Site VPN

Figure 1: Setup when connecting from on premises over Direct Connect or Site-to-Site VPN


When you enable private DNS for an S3 interface VPC endpoint, Amazon Web Services creates a private hosted zone and associates it with the VPC. This hosted zone includes the resource records for interface VPC endpoint with private IPs for each of the following Amazon S3 DNS names:

  • Regional Bucket (e.g., s3.<Region>.amazonaws.com)
  • Control (e.g., s3-control.<Region>.amazonaws.com)
  • Access Point (e.g., s3-accesspoint.<Region>.amazonaws.com)

This enables you to use Amazon Web Services’s private network connectivity to S3 while making requests to the service’s Regional, control, or access point endpoints. For more information, refer to the Amazon Web Services PrivateLink for Amazon S3 documentation .

Diagram walkthrough

  1. On-premises client initiates a DNS query targeted to a regional S3 bucket.
  2. The on-premises DNS server forwards this query to the respective Route 53 resolver inbound endpoint associated to the same VPC that has the S3 Interface VPC endpoint via the Site-to-Site VPN or DX connection.
  3. The Route 53 resolver endpoint forwards this query to the Route 53 hosted zone managed by Amazon Web Services which returns the IP addresses of the S3 Interface VPC endpoints in the DNS response.
  4. On-premises client then initiates the connection to the S3 interface VPC endpoint.
  5. S3 interface endpoints forwards the client query over Amazon Web Services PrivateLink towards the S3 bucket specified in client’s query.

New – Enable private DNS only for inbound endpoint

Many customers have applications on premises and in an Amazon Web Services Region, both running in the same VPC. These customers told us that they wanted an easy way to route traffic from on-premises through interface endpoints, and traffic from within Amazon Web Services over gateway endpoints. To solve this problem, we introduced the Enable private DNS only for inbound endpoint option. When you check enable DNS name for Amazon S3 interface endpoints, the Enable private DNS only for inbound endpoint option will be checked by default. In this case, the DNS queries for S3 originating from on-premises resolve to the private IPs of S3 interface endpoints, while S3 DNS queries originating within the VPC continue to resolve to public IPs of S3 using gateway VPC endpoint. For this configuration to work, you need to have a gateway endpoint present in your VPC. If you do not, and you try to enable this configuration, you will see an error like this in Amazon Web Services Management Console ( Figure 2 ):

“To set PrivateDnsOnlyForInboundResolverEndpoint to true, the VPC <vpce_id> must have a gateway endpoint for the service.”

To resolve this, create a gateway endpoint in the VPC. Alternatively, you can disable Enable private DNS only for inbound endpoint and route all your traffic over the interface endpoint.

Figure 2 Error when Gateway Endpoint does not exist while PrivateDNSonlyForInboundEndpoint enabled

Figure 2: Error when gateway endpoint does not exist while PrivateDNSonlyForInboundEndpoint enabled

Prerequisites

Before you get started, make sure you have the following prerequisites met:

  1. Create a VPC in the same Region as the S3 bucket that you wish to connect over the VPC endpoints . Make sure you have set the attributes enableDnsHostnames and enableDnsSupport to true.
  2. Create either an Amazon Web Services Direct Connect connection with a Private Virtual Interface (VIF) or an Amazon Web Services Site-to-Site VPN connection to establish connectivity from your corporate data center.
  3. Create a gateway VPC endpoint for S3 in the same VPC created in step 1 to use Enable private DNS only inbound endpoint .

Creating an interface VPC endpoint for Amazon S3 and enable private DNS options

  1. To create an interface VPC endpoint for Amazon S3, first navigate to the VPC console, select Endpoints , and choose Create endpoint .
  2. For Service category , select Amazon Web Services services . Then, filter the service names by entering S3 in the search box. For Service name , choose the service as “S3” and for Type , ensure that it shows Interface ( Figure 3a ).

Figure 3a: Create Interface endpoint for s3

Figure 3a: Create Interface endpoint for S3

  1. Select the VPC, desired Availability Zones, and subnet for each one, and then select the appropriate security groups. This should allow traffic from your networks on port 443.
  2. Under Additional settings , select Enable DNS name for your interface endpoint. By default, it will select Enable private DNS only for inbound endpoint to make traffic originating inside your VPC flow over the gateway VPC endpoint and traffic originating on premises flow over the interface VPC endpoint.
  3. Select Create endpoint . We show this in the following screenshot ( Figure 3b ):

Figure 3b: Select PrivateDNS options while Creating interface VPC endpoint in the VPC console

Figure 3b: Select private DNS options while creating interface VPC endpoint in the VPC console

It takes a few moments to go through various endpoint states while it is created. Once the interface endpoint status says available, you can view its configuration by choosing Details ( Figure 4a ). The DNS Names field displays the DNS names used to access the service. With private DNS enabled, you can see the default S3 Region DNS name as well.

Figure 4a: Details of the interface VPC endpoint

Figure 4a: Details of the interface VPC endpoint

Choose Subnets to see where the interface endpoint is located, and the ID of the endpoint network interface in each subnet. In the following screenshot ( Figure 4b ), the private IP address of the endpoint network interface in the VPC are 10.0.4.122 and 10.0.23.155.

Figure 4b: Subnets information for your interface VPC endpoint

Figure 4b: Subnets information for your interface VPC endpoint

Scenarios for private DNS options

Let’s understand the various combinations of DNS options that influence clients connectivity to Amazon S3 from applications hosted in VPC and on-premises using Gateway and Interface VPC endpoint for Amazon S3:

Scenario 1 : With out private DNS options

In this configuration, traffic from clients within the VPC where the gateway endpoint has been created can connect to S3 Regional endpoints. Clients outside the VPC (either on-premises or another interconnected VPC) can connect to S3 using the endpoint-specific DNS names or using the options highlighted here in the blog . In this case, you cannot set Enable private DNS only for inbound endpoint when private DNS is set to false.

This option is useful when you want to have the flexibility to manage private DNS names in your own private hosted zone.

Client in VPC using gateway endpoint:

dig s3.<Region>.amazonaws.com +short
54.x.y.z (Public IP address(es) of S3 service endpoint)

Client in VPC or on premises using endpoint-specific DNS name:

dig *.vpce-0cd6fd8a1a7d95f7e-4nyak8fx.s3.<Region>.vpce.amazonaws.com +short
10.0.23.155
10.0.4.122

The preceding output shows that clients inside the VPC resolve to a public IP address(es) of an S3 service endpoint, while the clients in the corporate data center resolve to interface VPC endpoint ENI IP addresses for Amazon S3.

Scenario 2 : With private DNS

In this configuration, both in-VPC and on-premises traffic flows over an interface VPC endpoint for S3. This option is beneficial when you want to simplify the architecture to just use one type of endpoint since it simplifies DNS management. However, this is not a cost-efficient solution since now the traffic from resources in VPC to S3 would also incur data transfer charges associated with interface VPC endpoint for S3. VPC As shown in Figure 5 , the green and blue color show that the traffic is flowing from the EC2 instances within the VPC and on-premises environment over the interface VPC endpoint for S3.

Figure 5 - With private DNS enabled and Enable private DNS only for inbound endpoint disabled

Figure 5: With private DNS enabled and Enable private DNS only for inbound endpoint disabled

Diagram walkthrough

All the Steps 1 through 5 remain the same as described in Figure 1 . However, now with only Private DNS enabled clients inside the VPC as well as the clients in on-premise connect to Amazon S3 via the interface VPC endpoint for S3.

Client inside the VPC:

dig s3.<Region>.amazonaws.com +short
10.0.23.155
10.0.4.122

Client inside on-premises application:

dig s3.<Region>.amazonaws.com +short
10.0.23.155
10.0.4.122

The preceding output shows that both the clients inside the VPC and on premises resolve to interface VPC endpoint ENI IP addresses for Amazon S3.

Scenario 3: With private DNS only for the inbound resolver endpoint

In this configuration, the traffic from applications within the VPC flows over gateway VPC endpoints, while on-premises traffic flows over the interface VPC endpoint for S3. This option provides a cost-effective network design to access S3 from within VPC and on-premises applications. While choosing this configuration, you need to maintain a gateway VPC endpoint in you VPC. This is to keep your traffic always on the Amazon Web Services private network. This eliminates the possibility where, without a gateway endpoint, you’re in-VPC traffic inadvertently goes over an Internet Gateway or is dropped if there is no Internet Gateway. Hence, if a gateway VPC endpoint does not exist in the VPC where you have applications running, it will prevent you from selecting the Enable private DNS only for inbound endpoint option . If you want to want to update an existing interface endpoint to Enable private DNS only for inbound endpoint , you must confirm that your VPC has a gateway VPC endpoint for S3. For more information about gateway VPC endpoints and managing private DNS names, see Gateway VPC endpoints and Manage DNS names respectively in the Amazon Web Services PrivateLink Guide.

Figure 6 shows the blue path for traffic from EC2 instances within the VPC flowing via gateway VPC endpoints, whereas the green path shows the traffic flow from on premises to S3 using interface VPC endpoint.

Figure 6 - With private DNS enabled and Enable private DNS only for inbound endpoint enabled

Figure 6: With private DNS enabled and Enable private DNS only for inbound endpoint enabled

Diagram walkthrough

All the Steps 1 through 5 remain the same as Figure 1 . However, now with both Private DNS and Enable private DNS only for inbound endpoint enabled, clients inside the VPC connect to the Amazon S3 via Gateway VPC endpoint for S3 while clients on-premise connect to Amazon S3 via the interface VPC endpoint for S3.

Clients in VPC using gateway endpoint:

$ dig s3.<Region>.amazonaws.com +short
54.x.y.z (Public IP address(es) of S3 service endpoint)

Clients in on premises using interface endpoint:

$ dig s3.<Region>.amazonaws.com +short
10.0.23.155
10.0.4.122

You cannot delete a gateway VPC endpoint when both private DNS and Enable private DNS only for inbound endpoint are enabled. If you attempt to do that, it will throw the following error:

“Gateway endpoint cannot be deleted while Interface endpoint for the service has PrivateDnsOnlyForInboundResolverEndpoint set to true.”

In this case, if you want to delete the gateway VPC endpoint, you need to modify your interface VPC endpoint and deselect the option Enable private DNS only for inbound endpoint.

Conclusion

In this blog post, we discussed using private DNS for an Amazon S3 interface VPC endpoint to access Amazon S3 without modifying on-premises applications. We covered using the Enable private DNS only for inbound endpoint option to optimize the network path to S3. These options allow you to advantage of the lowest-cost private network path without having to make code or configuration changes to your clients. To get started with Amazon Web Services PrivateLink for Amazon S3, visit this page .

For more information on Amazon Web Services PrivateLink for Amazon S3, you can refer to the following blogs:

  • Secure hybrid access to Amazon S3 using Amazon Web Services PrivateLink
  • Choosing Your VPC Endpoint Strategy for Amazon S3
  • Amazon Web Services Partners use Amazon Web Services PrivateLink to connect privately to Amazon S3
Sumiran Tandon

Sumiran Tandon

Sumiran is a Senior Product Manager at Amazon Web Services on the Amazon S3 team based in Seattle, WA. She focuses on building innovative, customer-centric products to enable customers solve their complex business challenges. She holds an MBA degree from Duke University.

Rohit Aswani

Rohit Aswani

Rohit is a Senior Specialist Solutions Architect focussed on Networking at Amazon Web Services, where he helps customers build and design scalable, highly-available, secure, resilient and cost effective networks. He holds a MS in Telecommunication Systems Management from Northeastern University, specializing in Computer Networking. In his spare time, Rohit enjoys hiking, traveling and exploring new coffee places.

Shubham Singh

Shubham Singh

Shubham is a Senior Network Specialist Solutions Architect at Amazon Web Services. He helps customers design innovative, resilient, and cost-effective solutions using Amazon Web Services services. He is passionate about technology and enjoys building solutions in the Networking and Security. He holds a MS in Telecommunication Systems Management from Northeastern University, specializing in Computer Networking.


The mentioned AWS GenAI Services service names relating to generative AI are only available or previewed in the Global Regions. Amazon Web Services China promotes AWS GenAI Services relating to generative AI solely for China-to-global business purposes and/or advanced technology introduction.