Amazon MSK Replicator now supports enhanced consumer offset synchronization for bidirectional replication
Amazon MSK Replicator now provides enhanced consumer offset synchronization for bidirectional replication, enabling applications to resume processing from the correct position when moving across Kafka clusters. This capability enables you to move producer and consumer applications between clusters independently, in any order, without the risk of losing data.
MSK Replicator is a feature of Amazon MSK that automates data replication between Kafka clusters, eliminating the need to manage custom replication infrastructure or configure open-source tools. Previously, while replicating bidirectionally with MSK Replicator, consumer group offsets were synchronized only when producers and consumers were active on the same cluster, requiring careful sequencing of application migrations between clusters and increasing the risk of duplicate message processing during rollbacks. With this launch, MSK Replicator synchronizes consumer group offsets across source and target clusters regardless of where producers are running, enabling applications to move between clusters without coordination constraints or data duplication risks.
You can enable enhanced consumer offset synchronization when creating a Replicator using the Amazon MSK console, Amazon CLI, or Amazon CloudFormation. This capability is supported in all Amazon Web Services Regions where MSK Replicator is available including Amazon Web Services China (Beijing) Region, operated by Sinnet and Amazon Web Services China (Ningxia) Region, operated by NWCD.
To learn more, visit the MSK Replicator documentation, product page, pricing page, and this Amazon Web Services blog post.