Posted On: Dec 20, 2019
We now support four failure-handling features for processing Kinesis and DynamoDB streams: Bisect on Function Error, Maximum Record Age, Maximum Retry Attempts, and Destination on Failure. These new features allow you to customize responses to data processing failures and build more resilient stream processing applications.
Lambda processes data records from Kinesis and DynamoDB streams in batches. Previously when your function returns an error, Lambda stops processing any data in the impacted shard and retries the entire batch of records. These records are continuously retried until they are successfully processed by Lambda or expired by the event source.
Bisect on Function Error
With Bisect on Function Error enabled, Lambda breaks the impacted batch of records into two when a function returns an error, and retries them separately. This allows you to easily separate the malformed data record from the rest of the batch, and process the rest of data records successfully.
Maximum Record Age
Your Lambda function can skip processing a data record when it has reached its Maximum Record Age, which can be configured from 60 seconds to 7 days.
Maximum Retry Attempts
Your Lambda function can skip retrying a batch of records when it has reached the Maximum Retry Attempts, which can be configured from 0 to 10,000.
Destination on Failure
Now your Lambda function can continue processing a shard even when it returns an error. When a data record reaches its Maximum Retry Attempts or Maximum Record Age, you can send its metadata like shard ID and stream ARN to one of these two destinations for further analysis: an SQS queue or SNS topic.
You can get started with these features with Amazon Web services Management Console, Amazon CLI, or Amazon SDK for Lambda. They can be used at no additional cost in all China Regions where Amazon Lambda is available. For more information on where Amazon Lambda is available in China, see the Amazon Web Services China Region table. To learn more, see Using Amazon Lambda with Amazon Kinesis and Using Amazon Lambda with Amazon DynamoDB in the Amazon Lambda Developer Guide, and this blog post.