Amazon MSK features

Amazon MSK is a fully managed service that makes it easy for you to build and run applications that use Apache Kafka to process streaming data. Managing Apache Kafka clusters is complex and time consuming. Amazon MSK makes it easy for you to build and run production applications on Apache Kafka without needing Apache Kafka infrastructure management expertise so you spend less time managing infrastructure and more time building applications.

Amazon MSK supports native Apache Kafka APIs and existing open-source tools built against those APIs. This enables existing Apache Kafka applications to work with Amazon MSK clusters without changes to application code. You continue to use Apache Kafka’s APIs and the open-source ecosystem to populate data lakes, stream changes to and from databases, and power machine learning and analytics applications.

With a few clicks in the Amazon MSK console, you can create a fully managed Apache Kafka cluster that follows Apache Kafka’s deployment best practices, or you can create your own cluster using your own custom configuration. Once you create your desired configuration, Amazon MSK automatically provisions, configures, and manages the operations of your Apache Kafka cluster and Apache ZooKeeper nodes.

Apache ZooKeeper is required to run Apache Kafka, coordinate cluster tasks, and maintain state for resources interacting with the cluster. Amazon MSK manages the Apache ZooKeeper nodes for you. Each Amazon MSK cluster includes the appropriate number of Apache ZooKeeper nodes for your Apache Kafka cluster at no additional cost.

Amazon MSK continuously monitors the health of your clusters and replaces unhealthy brokers without downtime for your applications. Amazon MSK manages the availability of your Apache ZooKeeper nodes so you will not need to start, stop, or directly access the nodes yourself. Amazon MSK also deploys software patches as needed to keep your cluster up to date and running smoothly.

Amazon MSK uses multi-AZ replication for high-availability. Data replication is included at no additional cost.  

Your Apache Kafka cluster runs in an Amazon Virtual Private Cloud (VPC) managed by Amazon MSK. Kafka clients in your own Amazon VPC can access the cluster privately through a cross-account elastic network interface that Amazon MSK deploys in your VPC. If your Kafka clients are spread across one or more VPCs or Amazon Web Services accounts, you can still connect privately to your cluster by using the multi-VPC private connectivity feature. This feature eliminates the operational overhead of self-managing a PrivateLink solution and scales seamlessly as the Amazon MSK cluster scales, enabling you to maintain private connectivity to the cluster without making additional configuration changes. Multi-VPC private connectivity also eliminates the challenges with managing non-overlapping IPs, complex peering and routing tables associated with other VPC connectivity solutions as it allows for overlapping IPs across connecting VPCs.

Amazon MSK encrypts your data at rest without special configuration or third-party tools. All data can be encrypted at rest using Amazon Key Management Service (KMS) Customer Master Key (CMK) by default, or your own CMK.

Amazon MSK also encrypts data in-transit via TLS between brokers and between clients and brokers on your cluster.

You can start with a few brokers within an Amazon MSK cluster. Then, using the Amazon Web Services management console or Amazon CLI, you can scale up to 100’s of brokers per cluster. Submit a limit increase request if you need more than 15 brokers per cluster or more than 30 brokers per account.

You can seamlessly scale up the amount of storage provisioned per broker to match changes in storage requirements using the Amazon Web Services management console or Amazon CLI.

Amazon MSK makes it easier for customers to build end-to-end solutions by providing native Amazon Web Services integrations out-of-the-box. You can encrypt data at rest using Amazon KMS, deploy Amazon MSK using code with Amazon CloudFormation, privately connect clients within an Amazon VPC to Amazon MSK, and leverage Amazon Identity and Access Management (IAM) for fine-grained service-level API control.

Amazon MSK deploys a best practice cluster configuration for Apache Kafka by default, and gives customers the ability to tune more than 30 different cluster configurations while supporting all dynamic and topic-level configurations. For more information, see Custom MSK Configurations in the documentation.

Use a cluster policy for your Amazon MSK cluster to define which IAM principals have cross-account permissions to set up private connectivity to your Amazon MSK cluster. When used with IAM client authentication, you can also use the cluster policy to granularly define Kafka data plane permissions for the connecting clients.

With tiered storage, you can store virtually unlimited data in MSK without the need to provision and manage storage capacity with tiered storage. You can enable tiered storage with a few clicks for new or existing clusters and pay for what you use. You can first store data in a performance optimized primary storage tier and let MSK automatically tier data into the new low-cost tier for longer retention. The feature is supported in Amazon China (Beijing) region, operated by Sinnet and Amazon China (Ningxia) region, operated by NWCD. To learn how to get started with MSK and Tiered Storage, visit our Amazon MSK Developer Guide.