Amazon MSK features

Amazon MSK is a fully managed service that makes it easy for you to build and run applications that use Apache Kafka to process streaming data. Managing Apache Kafka clusters is complex and time consuming. Amazon MSK makes it easy for you to build and run production applications on Apache Kafka without needing Apache Kafka infrastructure management expertise so you spend less time managing infrastructure and more time building applications.

Amazon MSK supports native Apache Kafka APIs and existing open-source tools built against those APIs. This enables existing Apache Kafka applications to work with Amazon MSK clusters without changes to application code. You continue to use Apache Kafka’s APIs and the open-source ecosystem to populate data lakes, stream changes to and from databases, and power machine learning and analytics applications.

With a few clicks in the Amazon MSK console, you can create a fully managed Apache Kafka cluster that follows Apache Kafka’s deployment best practices, or you can create your own cluster using your own custom configuration. Once you create your desired configuration, Amazon MSK automatically provisions, configures, and manages the operations of your Apache Kafka cluster and Apache ZooKeeper nodes.

Apache ZooKeeper is required to run Apache Kafka, coordinate cluster tasks, and maintain state for resources interacting with the cluster. Amazon MSK manages the Apache ZooKeeper nodes for you. Each Amazon MSK cluster includes the appropriate number of Apache ZooKeeper nodes for your Apache Kafka cluster at no additional cost.

Amazon MSK continuously monitors the health of your clusters and replaces unhealthy brokers without downtime for your applications. Amazon MSK manages the availability of your Apache ZooKeeper nodes so you will not need to start, stop, or directly access the nodes yourself. Amazon MSK also deploys software patches as needed to keep your cluster up to date and running smoothly.

Amazon MSK uses multi-AZ replication for high-availability. Data replication is included at no additional cost.  

Your Apache Kafka clusters run in an Amazon VPC managed by Amazon MSK. Your clusters are available to your own Amazon VPCs, subnets, and security groups based on the configuration you specify. You have complete control of your network configuration, and IP addresses from your VPCs are attached to your Amazon MSK resources through elastic network interfaces (ENIs).

Amazon MSK encrypts your data at rest without special configuration or third-party tools. All data can be encrypted at rest using AWS Key Management Service (KMS) Customer Master Key (CMK) by default, or your own CMK.

Amazon MSK also encrypts data in-transit via TLS between brokers and between clients and brokers on your cluster.

You can start with a few brokers within an Amazon MSK cluster. Then, using the AWS management console or AWS CLI, you can scale up to 100’s of brokers per cluster. Submit a limit increase request if you need more than 15 brokers per cluster or more than 30 brokers per account.

You can seamlessly scale up the amount of storage provisioned per broker to match changes in storage requirements using the AWS management console or AWS CLI.

Amazon MSK makes it easier for customers to build end-to-end solutions by providing native AWS integrations out-of-the-box. You can encrypt data at rest using AWS KMS, deploy Amazon MSK using code with AWS CloudFormation, privately connect clients within an Amazon VPC to Amazon MSK, and leverage AWS Identity and Access Management (IAM) for fine-grained service-level API control.

Amazon MSK deploys a best practice cluster configuration for Apache Kafka by default, and gives customers the ability to tune more than 30 different cluster configurations while supporting all dynamic and topic-level configurations. For more information, see Custom MSK Configurations in the documentation.