What's New in Amazon Kinesis

Amazon Kinesis Data Streams for Amazon DynamoDB now supports Amazon CloudFormation for DynamoDB global tables, which means you can enable streaming to an Amazon Kinesis data stream on your DynamoDB global tables with CloudFormation templates. By streaming your DynamoDB data changes to a Kinesis data stream, you can build advanced streaming applications with Amazon Kinesis services. For example, Amazon Kinesis Data Analytics reduces the complexity of building, managing, and integrating with Apache Flink and provides built-in functions to filter, aggregate, and transform streaming data for advanced analytics. You also can use Amazon Kinesis Data Firehose to take advantage of managed streaming delivery of DynamoDB table data to other Amazon Web Services services such as Amazon OpenSearch Service, Amazon Redshift, and Amazon S3.

Amazon Kinesis Data Streams is a serverless streaming data service that makes it easy to capture, process, and store streaming data at massive scale. Data Viewer is a new capability for Amazon Kinesis Data Streams that allows viewing data records directly from Amazon Web Services Management Console. As a result, you can easily inspect the data records without programming a dedicated consumer app just to view the data, quickly check the data structure of an unfamiliar stream, or query specific records for QA and troubleshooting. 

You can now deliver Amazon Virtual Private Cloud (VPC) Flow Logs directly to Amazon Kinesis Firehose, allowing you to stream your flow logs real-time to destinations supported by Amazon Kinesis Firehose or downstream logging solutions via custom HTTP endpoints.

Amazon Kinesis Data Firehose now supports Splunk as a delivery destination in Amazon Web Services China (Beijing) Region, operated by Sinnet, and Amazon Web Services China (Ningxia) Region, operated by NWCD. This native integration between Kinesis Data Firehose and Splunk Enterprise is designed to make data ingestion into Splunk Enterprise seamless, secure, and reliable. With this integration, there are no heavy forwarders to manage or set up. You just create a Kinesis Data Firehose delivery stream, stream events into it, and the delivery stream will continuously deliver these events to Splunk. You also have the option to backup your raw data to Amazon Simple Storage Service (Amazon S3).

Amazon Kinesis Data Firehose can now continuously partition streaming data by keys within data like “customer_id” or “transaction_id”, and deliver data grouped by these keys into corresponding Amazon Simple Storage Storage (Amazon S3) prefixes, making it easier for you to perform high performance, cost efficient analytics on streaming data in Amazon S3 using Amazon Athena, Amazon EMR, and Amazon Redshift Spectrum.

Amazon Kinesis Data Streams for Amazon DynamoDB now supports Amazon CloudFormation, which means you can enable streaming to an Amazon Kinesis data stream on your DynamoDB tables with CloudFormation templates. By streaming your DynamoDB data changes to a Kinesis data stream, you can build advanced streaming applications with Amazon Kinesis services. For example, Amazon Kinesis Data Analytics reduces the complexity of building, managing, and integrating with Apache Flink, and provides built-in functions to filter, aggregate, and transform streaming data for advanced analytics. You also can use Amazon Kinesis Data Firehose to take advantage of managed streaming delivery of DynamoDB table data to other Amazon Web Service services such as Amazon Elasticsearch Service, Amazon Redshift, and Amazon S3.

Amazon Kinesis Data Analytics for Apache Flink now supports UpdateApplicationMaintenanceConfiguration in preview. Amazon Kinesis Data Analytics periodically patches the underlying infrastructure of applications with OS and container-image security updates to meet compliance and security goals during the default maintenance windows in each region. You can use UpdateApplicationMaintenanceConfiguration via the CLI or the API to choose your preferred maintenance window start times for an application.

Amazon Kinesis Data Analytics for Apache Flink now provides access to the Apache Flink Dashboard giving you greater visibility to your applications and advanced monitoring capabilities. You can now view your Apache Flink application’s environment variables, over 120 metrics, logs, and the directed acyclic graph (DAG) of the application in a simple, contextualized user interface. 

You can now build and run streaming applications using Apache Flink version 1.11 in Amazon Kinesis Data Analytics for Apache Flink. Apache Flink v1.11 provides improvements to the Table and SQL APIs. The Table API is a unified, relational API for stream and batch processing and acts as a superset of the SQL language specially designed for working with Apache Flink.

You can now encrypt data in Amazon Kinesis Data Streams using server-side encryption. Data is encrypted using Amazon Key Management Service (KMS) managed customer master key. Server-side encryption makes it easy to meet strict data regulatory requirements, and enhances security by encrypting streaming data at rest within Amazon Kinesis Data Streams.  

With Amazon Kinesis Data Streams for Amazon DynamoDB, you can capture item-level changes in your DynamoDB tables as a Kinesis data stream. You can enable streaming to a Kinesis data stream on your table with a single click in the DynamoDB console, or via the Amazon API or Amazon CLI.  

Apache Flink Kinesis Consumer now supports Enhanced Fan Out (EFO) and the HTTP/2 data retrieval API for Amazon Kinesis Data Streams. EFO allows Amazon Kinesis Data Streams consumers to scale by offering each consumer a dedicated read throughput up to 2MB/second. The HTTP/2 data retrieval API reduces latency of data delivery from producers to consumers to 70 milliseconds or better. In combination, these two features allow you to build low latency Apache Flink applications that utilize dedicated throughput from Amazon Kinesis Data Streams.

Amazon Kinesis Data Analytics for Apache Flink now supports streaming applications built using Apache Beam Java SDK version 2.23. Apache Beam is an open-source, unified model for defining streaming and batch data processing applications that can be executed across multiple execution engines. This release allows you to build Apache Beam streaming applications in Java and run them using Apache Flink 1.8 on Amazon Kinesis Data Analytics, Apache Spark running on-premises, and other execution engines supported by Apache. 

You can now process multiple Amazon Kinesis data streams with a single Kinesis Client Library (KCL) based consumer application. Previously, each KCL based application processes a single Kinesis data stream. With this new capability, you can update the list of streams at runtime for multi-stream processing in a scalable KCL application without redeploying the application. This reduces operational overhead of maintaining multiple KCL applications. 

Amazon Kinesis Data Analytics now allows you to Force Stop a running application and reset its status to Ready, giving you more control over your applications. In addition, you can check if your application is scaling up or down through the new ‘Autoscaling' application status. 

With v2.0.0 of the Apache Flink Kinesis Data Firehose Producer, Amazon Kinesis Data Analytics now enables Apache Flink applications to send data directly to Amazon Kinesis Data Firehose. The new version adds support for the Amazon Identity and Access Management (IAM) Assume Role credentials provider, includes bug fixes, and updates the Amazon SDK and other dependencies.  

Amazon Kinesis Data Streams announces two new API features that simplify applications that consume data from Kinesis Data streams. The first feature is an optional filtering field in the ListShards API that enables you to set filtering criteria for the shards that you want to list from a data stream. For example, you can now use a timestamp filter to list shards with data at a certain time point. The second feature is a new field called ChildShards in the GetRecords and the SubscribeToShards APIs that enables you to discover child shards of a closed shard (without having to use the ListShards API to list all closed and open shards). Applications consuming data from Kinesis Data Streams can use this new field to efficiently transition reading from a closed shard to reading from its child shard. This is a common use case encountered by consuming applications when a shard is closed due to stream scaling operations.  

Amazon Kinesis Data Analytics is now available in the Amazon Web Services China (Beijing) Region, operated by Sinnet and the Amazon Web Services China (Ningxia) Region, operated by NWCD.  

Amazon Kinesis Data Firehose can now deliver streaming data to an Amazon Elasticsearch Service domain in an Amazon VPC. You can now easily ingest, transform, and reliably deliver streaming data in to an Amazon Elasticsearch Service domain running in a VPC without building and managing your own data ingestion and delivery infrastructure. 

Amazon Kinesis Data Firehose can now deliver data to Amazon Redshift in Amazon Web Services China (Beijing) Region, operated by Sinnet, and Amazon Web Services China (Ningxia) Region, operated by NWCD. You can now specify Amazon Redshift as a destination for your Amazon Kinesis Data Firehose delivery streams with a few clicks, without having to manage and scale your ingestion and delivery infrastructure. You can then use your existing analytics applications and tools to analyze streaming data in near real time. 

Amazon Kinesis Data Firehose is now available in the Amazon Web Services China (Beijing) Region, operated by Sinnet, and Amazon Web Services China (Ningxia) Region, operated by NWCD. Amazon Kinesis Data Firehose is the easiest way to load streaming data into data stores and analytics tools. It can capture, transform, and load streaming data into Amazon S3 and Amazon Elasticsearch Service, enabling near real-time analytics with existing business intelligence tools you are already using today.

We are excited to announce the new ListShards API which makes it easier for your Kinesis Data Streams KCL applications to scale. Many Kinesis Data Streams customers use the KCL to process records from their data streams. Starting in version 1.9.0 or later, key KCL API functions now use the ListShards API, which supports a 10x higher call rate limit compared to the DescribeStream API, which was used for these functions in earlier KCL versions.