New: Improved flexibility when configuring endpoint URLs with the Amazon Web Services SDKs and tools

The Amazon Web Services SDKs and Tools team is excited to announce improvements for configuration of the endpoint URL used for API service requests through the shared SDK configuration file and environment variables with the Amazon Web Services SDKs and Tools .

Previously, you could specify the endpoint URL used for Amazon Web Services requests by setting the --endpoint-url command line parameter when invoking the Amazon Web Services CLI or supplying an endpoint to an SDK client constructor. When switching between testing and production environments like Amazon DynamoDB local and the Amazon DynamoDB web service , you must provide different endpoints directly. Now, you can configure the endpoint URL using the same mechanisms as other Amazon Web Services CLI or SDK configuration parameters. In addition, you can specify the endpoint to be used for all Amazon Web Services services or for a specific Amazon Web Services service.

This post demonstrates how to configure the endpoint URL when using the local implementations of Amazon DynamoDB local and Amazon Web Services Step Functions Local through the Amazon Web Services Command Line Interface (Amazon Web Services CLI) .

Prerequisites

In order to use this feature with the Amazon Web Services CLI, you must have Amazon Web Services CLI 1.29.0 or greater, or 2.13.0 or greater, installed. To install the Amazon Web Services CLI, follow the instructions in the “Install or update the latest version of the Amazon Web Services CLI” page of the Amazon Web Services CLI User Guide.

This post makes use of Docker container images for the local implementations of Amazon DynamoDB local and Amazon Web Services Step Functions Local . This post assumes that you can run these Docker container images. See the respective reference documentation pages for each of these services to learn more about how to run these services locally.

Configure the endpoint URL for all API requests

First, we demonstrate how to set the endpoint URL that will be used for all API requests made with the Amazon Web Services CLI. The Amazon DynamoDB local implementation can be started in the following way to open port 8080 to accept API requests and direct them to port 8000 inside of the Docker container:

$ docker run --rm -p 8080:8000 amazon/dynamodb-local

Note: the --rm flag indicates to clean up the container when the command exits. By default, a container’s file system persists even after the container exits.

You can connect to this port using the URL http://localhost:8080. You can use this endpoint URL with the Amazon Web Services CLI’s aws dynamodb commands to send requests to the Amazon DynamoDB local service running in the Docker container.

Previously, you would only be able to use this endpoint URL with the --endpoint-url parameter for the Amazon Web Services CLI. Let’s create a table in our local implementation using the --endpoint-url parameter:

$ aws --endpoint-url http://localhost:8080/ dynamodb create-table \
  --table-name MyLocalTable \
  --attribute-definitions AttributeName=ColumnA,AttributeType=S AttributeName=ColumnB,AttributeType=S \
  --key-schema AttributeName=ColumnA,KeyType=HASH AttributeName=ColumnB,KeyType=RANGE \
  --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

Using environment variables

With the new feature, you can set the endpoint via the environment variable AWS_ENDPOINT_URL, and now you can list your newly created MyLocalTable table:

$ export AWS_ENDPOINT_URL=http://localhost:8080/
$ aws dynamodb list-tables
{
    "TableNames": [
        "MyLocalTable"
    ]
}

To stop using the AWS_ENDPOINT_URL configured endpoint URL, you must unset it in your environment:

$ unset AWS_ENDPOINT_URL

Using the SDK shared configuration file

This new feature also allows you to set the endpoint URL in a profile in the SDK shared configuration file. Add the following profile to your shared configuration file (by default, this file is at ~/.aws/config):

[profile local]
endpoint_url=http://localhost:8080/

Now you can use the profile local in the Amazon Web Services CLI command and put an item in the MyLocalTable table:

$ aws --profile local dynamodb list-tables
{
    "TableNames": [
        "MyLocalTable"
    ]
}

Configure the endpoint URL for specific services

A limitation of setting the endpoint URL through the AWS_ENDPOINT_URL environment variable or endpoint_url profile property is that all API requests, including those to other Amazon Web Services services will be directed to this endpoint URL. This might be convenient if you are only using a single Amazon Web Services service, or if you are using a third-party local development environment like LocalStack.

However, you might want more control over which Amazon Web Services services use a different endpoint URL. To only send DynamoDB API requests to the endpoint, you can configure service-specific endpoint URLs through environment variables and the SDK shared configuration file.

Using environment variables

To set a service-specific endpoint URL for Amazon DynamoDB API requests, use the AWS_ENDPOINT_URL_DYNAMODB environment variable and retrieve the item we previously inserted:

$ AWS_ENDPOINT_URL_DYNAMODB=http://localhost:8080/ aws dynamodb list-tables
{
    "TableNames": [
        "MyLocalTable"
    ]
}

This technique is applicable to any Amazon Web Services service. A list of the available can be found in the “Identifiers for service-specific endpoints“ documentation.

Using the SDK shared configuration file

You can also now configure a service-specific endpoint URL in a profile in the SDK shared configuration file. To do this, you will use a services section and associate it with a profile. A services section contains sub-sections for the Amazon Web Services service(s) that you want to configure. The services section is referenced in a profile through the services profile parameter.

This technique is applicable to any Amazon Web Services service. A complete list of the available service section names can be found in the “Identifiers for service-specific endpoints“ documentation.

For example, to configure the Amazon Web Services DynamoDB endpoint to the local Docker container, add the following profile and services section to your shared configuration file:

[profile production]
region=us-west-2

[profile local]
region=us-west-2
services=local-services

[services local-services]
dynamodb =
  endpoint_url=http://localhost:8080/

Now you can use the profile local in the Amazon Web Services CLI command to make requests to the Amazon DynamoDB local installation:

$ aws --profile local dynamodb list-tables
{
    "TableNames": [
        "MyLocalTable"
    ]
}

Then, with no other configuration changes, you can specify the production profile to make requests to the Amazon DynamoDB web service:

$ aws --profile production dynamodb list-tables

Note: the output of this command depends if you have created any tables in the Amazon DynamoDB web service.

This enables you to keep the configuration that determines which endpoint URL to use outside of your scripts and in an officially supported configuration location.

For details on the syntax of the shared configuration file when specifying a custom endpoint URL, see the section “Configure endpoints using the shared config file” in the “Service-specific endpoints“ documentation.

Configure the endpoint URL for multiple services

You can also use the Amazon Web Services CLI with multiple Amazon Web Services services. This new feature allows you to configure separate custom endpoint URLs for each service at the same time. A list of the environment variable and service section names can be found on the “Identifiers for service-specific endpoints” documentation page. For example, to use Amazon Web Services Step Functions local at the same time as Amazon DynamoDB local, you can use the environment variable AWS_ENDPOINT_URL_SFN or services section sfn to set the endpoint when using aws stepfunctions commands.

The Amazon Web Services Step Functions local implementation can be started with the following command:

$ docker run --rm -p 8083:8083 amazon/aws-stepfunctions-local

Similar to the example for Amazon DynamoDB local, you can connect to this port using the URL http://localhost:8083. You can use this endpoint URL with the Amazon Web Services CLI’s aws stepfunctions commands to send requests to the Amazon Web Services Step Functions local service running in the Docker container.

To use both local implementations, you can set both environment variable at the same time to their respective endpoint URLs:

$ export AWS_ENDPOINT_URL_DYNAMODB=http://localhost:8080/
$ export AWS_ENDPOINT_URL_SFN=http://localhost:8083/

Then create an Amazon Web Services Step Functions state machine using our local implementation:

$ aws stepfunctions create-state-machine \
  --name HelloWorldLocal \
  --definition '{"StartAt": "HelloWorld", "States": {"HelloWorld": {"Type": "Pass", "Result": "Hello World", "End": true}}}' 
  --role-arn "arn:aws:iam::123456789012:role/HelloWorldLocalRole"
{
    "stateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:HelloWorldLocal",
    "creationDate": "2023-07-12T13:48:26.507000-07:00"
}

Now, list the tables and state machines:

$ aws dynamodb list-tables  # Uses the endpoint from AWS_ENDPOINT_URL_DYNAMODB
{
    "TableNames": [
        "MyLocalTable"
    ]
}
$ aws stepfunctions list-state-machines  # Uses the endpoint from AWS_ENDPOINT_URL_SFN
{
    "stateMachines": [
         {
            "stateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:HelloWorldLocal",
            "name": "HelloWorldLocal",
            "type": "STANDARD",
            "creationDate": "2023-07-12T13:48:26.507000-07:00"
        }
    ]
}

As before, unset the environment variables to stop using the configured endpoint URLs:

$ unset AWS_ENDPOINT_URL_DYNAMODB
$ unset AWS_ENDPOINT_URL_SFN

You can also define both endpoints in the SDK shared configuration file and specify the profile when calling the Amazon Web Services CLI. Add the following profile and services section to your shared configuration file:

[profile local]
region=us-west-2
services=local-services

[services local-services]
dynamodb =
  endpoint_url=http://localhost:8080/
sfn =
  endpoint_url=http://localhost:8083/

Invoking the Amazon Web Services CLI with --profile local will use the specified Amazon DynamoDB and Amazon Web Services Step Functions endpoint URLs for aws dynamodb and aws stepfunctions commands, respectively:

$ aws dynamodb list-tables --profile local
{
    "TableNames": [
        "MyLocalTable"
    ]
}
$ aws stepfunctions list-state-machines --profile local
{
    "stateMachines": [
        {
            "stateMachineArn": "arn:aws:states:us-east-1:123456789012:stateMachine:HelloWorldLocal",
            "name": "HelloWorldLocal",
            "type": "STANDARD",
            "creationDate": "2023-07-12T13:48:26.507000-07:00"
        }
    ]
}

Order of precedence for configuring the endpoint URL

You can configure the endpoint URL using any of the above methods at the same time. The endpoint URL used is determined by the Amazon Web Services CLI with the following order of precedence:

1. The value provided from the --endpoint-url command line parameter.
2. The value provided by a service-specific environment variable (like AWS_ENDPOINT_URL_DYNAMODB).
3. The value provided by the global endpoint environment variable (AWS_ENDPOINT_URL).
4. The value provided by the endpoint_url parameter from a services definition section in the shared configuration file.
5. The value provided by the endpoint_url parameter from a profile in the shared configuration file.

If an endpoint URL is not configured in any of these ways, the Amazon Web Services CLI uses the default endpoint URL for the respective Amazon Web Services service.

For more information about the order of precedence for Amazon Web Services SDK parameters, see the section “Precedence of settings” on the “Service-specific endpoints” documentation page.

Ignoring configured endpoint URLs

If you want to ignore configured endpoint URLs from the SDK shared configuration file or environment variables, you can use the profile parameter ignore_configured_endpoint_urls=true or the environment variable AWS_IGNORE_CONFIGURED_ENDPOINT_URLS=true. Note that setting either of these parameters will not prevent the use of endpoints provided to the --endpoint-url command line parameter. To read more about how to disable this feature, see the “Service-specific Endpoints” documentation page.

Cleaning up

If you’ve started any Docker containers, be sure to shut them down by terminating the process or closing the terminal where they were running when you’re done working through the examples in this post. Also be sure to either close the terminal window you were using or unset any environment variables used in this post:

$ unset AWS_ENDPOINT_URL
$ unset AWS_ENDPOINT_URL_DYNAMODB
$ unset AWS_ENDPOINT_URL_SFN

Next steps

In this post, we have demonstrated how to simplify the configuration of endpoint URLs when using the Amazon Web Services CLI. These examples only describe one way to use this feature. Other use cases are to configure Amazon Web Services Security Token Service (Amazon Web Services STS) VPC endpoints , S3-compatible object stores, or third-party local Amazon Web Services development environments like LocalStack.

The Amazon Web Services SDKs and tools that currently support this feature are the SDK for .NET , the SDK for Python , the SDK for Ruby , the Amazon Web Services CLI v1 , the Amazon Web Services CLI v2 , and the Amazon Web Services Tools for Powershell . To learn when other SDKs support this feature, see the “Compatibility with Amazon Web Services SDKs” section on the the “Service-specific endpoints” SDK documentation page.

For more information, see the “Service-specific endpoints” SDK documentation page. For any issues or feature requests, please file an issue on our GitHub repository . We look forward to hearing your feedback .

About the author: