Posted On: Mar 10, 2022
Amazon Lambda now allows customers to build applications that can be triggered by messages in an Apache Kafka cluster hosted on any infrastructure. Customers can keep their existing Apache Kafka cluster as-is, and quickly and easily build Kafka consumer applications with Lambda without needing to worry about provisioning or managing servers.
Today, Lambda supports Amazon Managed Streaming for Kafka (Amazon MSK) as an event source. Now, in addition to Amazon MSK, customers can also build applications triggered from Apache Kafka clusters that they manage, as long as Lambda can access the cluster via a public IP address or an Amazon VPC endpoint. Messages are processed in batches of up to 10,000 records, which may contain messages from one or more partitions. Messages within a partition are always processed in order. Lambda is invoked when the batch size is reached, or the payload size reaches 6MB. Lambda supports consuming messages from unauthenticated and authenticated clusters.
Customers can get started by selecting Apache Kafka as event source for their Lambda function through the Amazon Web Services Management Console, Amazon CLI or Amazon SDK for Lambda. This functionality is available in Amazon Web Services China (Beijing) Region, operated by Sinnet, and Amazon Web Services China (Ningxia) Region, operated by NWCD. This feature requires no additional charge to use. To learn more about building an Apache Kafka consumer application using Lambda, read the Lambda Developer Guide.