Posted On: Dec 19, 2022
Amazon Lambda now supports custom Consumer Group IDs when using Amazon Managed Streaming for Apache Kafka (Amazon MSK) or Self-Managed Kafka as an event source. Kafka uses Consumer Group IDs to identify consumer membership and record consumer checkpoints. Using a custom Consumer Group ID is ideal for customers with workloads that require disaster recovery or fail‑over support.
Lambda makes it easy to consume events from Kafka Topics at scale. When Lambda starts consuming from a topic, it presents a Consumer Group ID, this has always been a randomly generated unique value which ensures that Lambda will be identified as a new group consumer to the topic and that processing will start at the specified position (Latest or Trim horizon). Now, with a specified Consumer Group ID, Lambda does not need to be identified as a new consumer group. When Kafka identifies Lambda as an existing consumer group, consuming will instead start from where Kafka recorded the consumer group left off. In disaster recovery workflows, customers using Apache MirrorMaker2 can use Lambda with a custom Consumer Group ID to resume processing from a mirrored Kafka cluster.
You can get started with custom Consumer Group IDs for Amazon MSK and Self-Managed Kafka via Amazon Web Services Management Console, Amazon CLI, Amazon SAM, or Amazon SDK for Lambda. It can be used at no additional cost in Amazon Web Services China (Beijing) Region, operated by Sinnet, and Amazon Web Services China (Ningxia) Region, operated by NWCD. To learn more about building an Apache Kafka consumer application with Lambda, read the Lambda Developer Guide.