Filter Messages Published to Topics

with Amazon Simple Queue Service (Amazon SQS) and Amazon Simple Notification Service (Amazon SNS)

In this tutorial, you will learn how to leverage the message filtering feature of Amazon Simple Notification Service (SNS). The message filter feature enables endpoints subscribed to an Amazon SNS topic to receive only the subset of topic messages it is interested in.

In our example, users visit a website to place insurance quote requests. These quote requests can be for car insurance, boat insurance, or life insurance. When an insurance quote request is placed, the request is forwarded to a series of backend systems. There’s a sales system that handles vehicle insurance for cars and boats, another sales system that handles life insurance, and an analytics system that needs to process all quote requests.

As we learned in the Send Fanout Event Notifications Tutorial, we will use a fanout messaging pattern using Amazon SNS and Amazon SQS to decouple the website from the backend systems. To get the event notifications to the right backend system, you could create a separate topic for each type of quote request, then add message routing logic to your publisher. However, this option can result in overly complicated publishers, topic proliferation, and additional overhead in provisioning and managing your Amazon SNS topics. Amazon SNS message filtering is much simpler!

Manage Your Amazon Web Services Resources

Step 1. Create an Amazon SNS Topic

Your first step will be to create an Amazon SNS topic to publish messages whenever quote requests are placed on an insurance website. These leads can be related to car insurance, boat insurance, or life insurance.

a. When you click here, the Amazon Web Services Management Console will open in a new browser window, so you can keep this step-by-step guide open. When the screen loads, enter your user name and password to get started. Then type notification in the search bar and select Simple Notification Service to open the service console.

b. If the Amazon SNS console landing page appears, click Get Started Now. If you don't see this page, skip to the next step.

c. Under Common actions, click Create topic.

d. The Create new topic dialog box appears.

In the Topic name field, type Insurance-Quote-Requests, then click Create topic.

e. The new topic appears on the Topic details page.

Step 2. Create the Amazon SQS Queues

Each insurance sales system polls quote requests from its corresponding Amazon SQS message queue that is subscribed to the Amazon SNS topic.

The first queue will handle both car and boat insurance quote requests; the second queue will handle life insurance ones; and the third queue will catch all. The third queue will catch all quote requests, regardless of the insurance type, in order to feed an analytics system. These preferences will be set as filter policies on the Amazon SNS subscriptions.

In this step, you will create the three queues and subscribe each of them to the Amazon SNS topic.

a. Click here to open the Amazon SQS console in a new browser window. If the Amazon SQS landing page appears, click Get Started Now. Otherwise, proceed to the next step.

b. On the Create New Queue page, enter Vehicle-Insurance-Quotes in the Queue Name field.

Leave Standard Queue selected and click on Quick-Create Queue.

c. Click Create New Queue and repeat Step 2b to create two additional queues:
Life-Insurance-Quotes and All-Quotes.

d. All three queues are now listed in the Amazon SQS Console.

Step 3. Subscribe the Queues to the Topic

Now that you have created your three Amazon SQS queues, you need to subscribe them to the Amazon SNS topic that broadcasts new insurance quote requests.

a. From the list of queues, select the Vehicle-Insurance-Quotes, Life-Insurance-Quotes and All-Quotes queues.

Click Queue Actions and select Subscribe Queues to Amazon SNS Topic.

b. The Subscribe to a Topic dialog box is displayed. From the Choose a Topic drop-down list, select your Insurance-Quote-Requests topic, which you want to subscribe your three queues to. Because you created the Amazon SNS topic from the same account, it shows up in the list.

If the Amazon SNS topic was made by another account, you could subscribe to it by using the Topic ARN field. For more details, see the Amazon SNS documentation or our blog, Cross-Account Integration with Amazon SNS.

c. Leave the Topic Region unchanged, and click Subscribe.

d. The Topic Subscription Result dialog box is displayed. Click OK.

Step 4. Set Filter Policies to the Amazon SNS Subscriptions

Now that all Amazon SNS and Amazon SQS resources have been created, you are ready to set filter policies to your Amazon SNS subscriptions. A filter policy is simple JSON document, set as an attribute of the Amzon SNS subscription, which defines the type of notification the subscriber is interested in.

Subscriptions without filter policies will receive all messages published to Amazon SNS topic, so you don't need to set a filtering policy for the All-Quotes queue for this scenario.

a. Return to the Amazon SNS console in your other browser window, and click on Topics to find a list of all your topics in the region.

Click on the topic ARN for Insurance-Quote-Requests in the list.

b. This will take you to the Topic details page where you will be able to see all Amazon SQS queues subscribed to this Amazon SNS topic.

c. Select the subscription related to the Vehicle-Insurance-Quotes queue.

From Other subscription actions, click on Edit subscription filter policy.

d. On the dialog box displayed, enter the following JSON object:
{"insurance_type": ["car", "boat"]}

Click Set subscription filter policy.

e. A confirmation dialog appears that confirms the subscription filter policy was successfully created.

f. Now you will add the filter policy to the second queue. Select the Life-Insurance-Quotes queue, click Other subscription actions, and choose Edit subscription filter policy.

On the dialog box displayed, enter the following JSON object:
{"insurance_type": ["life"]}

Click Set subscription filter policy.

You don’t need to set any filter policy for the subscription related to the All-Quotes queue. This lack of policy will make this queue catch all events published to the topic.

Step 5. Publish Messages to the Topic

Let’s test it out! In this step, you will be able to simulate new insurance quote requests coming in from your lead generation website by publishing messages to the topic.

You will publish messages with different attributes, so you can view the message filtering mechanism of Amazon SNS in action.

a. While still on the Topic details page, click on Publish to topic.

b. The Publish a message page is displayed. Create the first message by entering the following values in the form:

In Subject, enter: Insurance Quote Request #1

In Message format, leave Raw selected

In Message, enter the following text to represent a car insurance quote request:
2017 Volvo S60, Montreal

In Message attributes:

Enter insurance_type in the key field
Select String in the Attribute type field
Enter car in the value field

Click Publish message.

c. You are returned to the Topic details page. Click on Publish to topic again, and create a second message using the following values:

In Subject, enter: Insurance Quote Request #2

In Message format, leave Raw selected

In Message, enter the following text to represent a life insurance quote request:
Male, 33 years old, Vancouver

In Message attributes:

Enter insurance_type in the key field
Select String in the Attribute type field
Enter life in the value field

Click Publish message.

d. You are returned to the Topic details page once more. To create your third and final message, click on Publish to topic and enter the following values:

In Subject, enter: Insurance Quote Request #3

In Message format, leave Raw selected

In Message, enter the following text to represent a home insurance quote request:
Townhouse, 1500 sq ft, Toronto

In Message attributes:

Enter insurance_type in the key field
Select String in the Attribute type field
Enter home in the value field

Click Publish message.

Step 6. Verify Your Message Deliveries

It’s time for you to verify that each type of insurance quote request was routed to its corresponding Amazon SQS message queue, by inspecting the message deliveries that took place.

You will see that a subscribing Amazon SQS queue has gotten a message only if its corresponding Amazon SNS subscription had a filter policy that matched the attributes of this incoming message.

You will also notice that one queue in particular actually received all messages published to the topic. This happened because this subscription had no filter policy set, which resulted in a catch-all behavior.

a. Return to the Amazon SQS console. Select the Vehicle-Insurance-Quotes queue, and click on Queue Actions.

Choose View/Delete Messages.

b. Click Start Polling for Messages.

Note that only one message was pushed into this queue, related to that car insurance quote (Insurance Quote Request #1). Close the dialog box.

c. Repeat the process for the second queue. Check the box for the Life-Insurance-Quotes queue. From the Queue Actions drop-down list, select View/Delete Messages.

Click Start Polling for Messages.

Note that another message was pushed into this queue, related to that life insurance quote (Insurance Quote Request #2).

Click Close.

d. Finally, check the box for the All-Quotes queue. Click Queue Actions, select View/Delete Messages, and click Start Polling for Messages.

Note that all three messages were pushed into this queue (Insurance Quote Requests #1, #2 and #3).

Click Close.

Step 7. Delete Your Resources

In this step, you will delete the resources you have created for this tutorial, which include a topic, subscriptions and queues. It is a best practice to delete resources you are no longer using, so you don’t incur charges.

a. In the Amazon SNS console, click Topics, then click on the ARN link related to topic Insurance-Quote-Requests.

b. On the Topic details page, select all three subscriptions.

Click Other subscription actions, then click Delete subscriptions.

Click Delete in the dialog box to confirm the deletion.

c. On the Topic details page, click on Other topic actions, then choose Delete topic.

Confirm the deletion on the dialog box.

d. Return to the Amazon SQS console. Select all three queues used in this tutorial, click Queue Actions, then click Delete Queues.

Confirm the deletion on the dialog box.

e. You can now sign out of the Amazon Web Services Management Console.

Congratulations!

You can now use message filtering on Amazon Simple Notification Service (SNS) to build simpler and more streamlined pub/sub architectures.

Message filtering enables Amazon SNS topic subscribers to selectively receive only a subset of the messages they are interested in, as opposed to receiving all messages published to a topic. This removes the design complexity of implementing message filtering logic in your subscribers, or message routing logic in your publisher systems.