Posted On: Mar 10, 2022
Amazon Lambda has launched improved auto scaling for Amazon MSK and self-managed Kafka as event sources to improve performance and help lower costs for customers. Lambda starts with one consumer and checks the OffsetLag metric (measure of backlog at source) and processing rate every minute and scales up or down every 3 minutes. Lambda allows up to one consumer per partition per topic for a Kafka cluster. Previously, Lambda used to start connection with Kafka clusters at a single consumer, then every 15 minutes would check for OffsetLag and scale up (or down) consumers.