Amazon Kinesis Data Analytics features

Amazon Kinesis Data Analytics is the easiest way to analyze streaming data in real time. Using templates and built-in operators, you can quickly and easily build queries and sophisticated real-time applications. Amazon Kinesis Data Analytics sets up the resources to run your applications and scales automatically to handle any volume of incoming data.

You do not need to setup and manage a complex infrastructure for high availability and stateful processing. Amazon Kinesis Data Analytics is serverless and takes care of everything required to continuously run your application. This includes automatically provisioning the infrastructure to continuously process streaming data.

Amazon Kinesis Data Analytics elastically scales applications to keep up with any volume of data in the incoming data stream. You only pay for the resources used to run your streaming applications. You need not worry about provisioning infrastructure or paying for idle capacity.

Amazon Kinesis Data Analytics delivers sub-second processing latencies so you can generate real-time alerts, dashboards, and actionable insights.

Amazon Kinesis Data Analytics includes open source libraries such as Apache Flink, Amazon SDK, and Amazon Web Services service integrations. Apache Flink is an open source framework and engine for building highly available and accurate streaming applications with support for Java, Python, SQL, and Scala. The Amazon SDKs help take the complexity out of coding for many Amazon Web Services services by providing APIs in your preferred language and includes the Amazon Web Services libraries, code samples, and documentation.

Flexible APIs are provided that are specialized for different use cases including stateful stream processing, streaming ETL, and real-time analytics. Pre-built operators and analytics capabilities enable you to build an Apache Flink streaming application in hours instead of months. The Amazon Kinesis Data Analytics libraries are extensible, enabling you to perform real-time processing for a wide variety of use cases.

You can setup and integrate a data source or destination with minimal code. You can use the Amazon Kinesis Data Analytics libraries to integrate with Amazon S3, Amazon Managed Streaming for Apache Kafka (Amazon MSK), Amazon OpenSearch Service (successor to Amazon Elasticsearch Service), Amazon DynamoDB, Amazon Kinesis Data Streams, Amazon Kinesis Data Firehose, and Amazon CloudWatch.

In addition to the Amazon Web Services integrations, the Amazon Kinesis Data Analytics libraries include more than ten connectors from Apache Flink and the ability to build custom integrations. With a couple more lines of code, you can modify how each integration behaves with advanced functionality. Also, you can build custom integrations using a set of Apache Flink primitives that enable you to read and write from files, directories, sockets, or other sources that you can access over the Internet.

You can use Apache Flink in Amazon Kinesis Data Analytics to build applications whose processed records affect the results exactly once, referred to as exactly once processing. This means that even in the case of an application disruption, like internal service maintenance or user initiated application update, the service will ensure that all data is processed and there is no duplicate data.

The service stores previous and in-progress computations, or state, in running application storage. This enables you to compare real-time and past results over any time period and provides fast recovery during application disruptions. State is always encrypted and incrementally saved in running application storage.

You can create and delete durable application backups through a simple API call. You can immediately restore your applications from the latest backup after a disruption, or you can restore your application to an earlier version.

Kinesis Data Analytics Studio supports sub-second queries with built-in visualizations. You can perform ad-hoc queries to quickly inspect your data stream and view results in seconds.

Studio notebooks provide a single-interface development experience for developing, debugging code, and running stream processing applications. 

Kinesis Data Analytics Studio supports SQL, Python, and Scala in the same development environment. Syntax highlighting, validation, and context-sensitive suggestions guide you within the notebook to interact with your data with built-in support for Apache Flink specific capabilities.

There are no servers to provision, manage, or scale, just write code, and pay for the resources your applications consume. Easily deploy your code in the notebook to a continuously running stream processing application with autoscaling and durable state.

Kinesis Data Analytics Studio runs on and produces Apache Flink applications used in production and Apache Zeppelin notebooks provide a familiar, easy to use experience for authoring streaming applications in a language of choice.

Amazon Glue Data Catalog is a persistent metadata store that serves as a central repository containing table definitions. You can use the Amazon Glue Data Catalog to quickly discover and search across multiple datasets. Amazon Kinesis Data Analytics Studio is compatible with the Amazon Glue Data Catalog where you can define the schema for your source and destination tables.

Amazon Kinesis Data Analytics supports standard ANSI SQL, so all you need is familiarity with SQL.

Amazon Kinesis Data Analytics integrates with Amazon Kinesis Data Streams and Amazon Kinesis Data Firehose so that you can readily ingest streaming data. Just point Amazon Kinesis Data Analytics at the input stream and it will automatically read the data, parse it, and make it available for processing. You can emit processed results to other Amazon Web Services services including Amazon S3, Amazon Redshift, and Amazon OpenSearch Service through Amazon Kinesis Data Firehose. You can also send output data to Amazon Kinesis Data Streams to build advanced stream processing pipelines.

You get an interactive editor to build SQL queries using streaming data operations like sliding time-window averages. You can also view streaming results and errors using live data to debug or further refine your script interactively.

Amazon Kinesis Data Analytics provides an easy-to-use schema editor to discover and edit the structure of the input data. The wizard automatically recognizes standard data formats such as JSON and CSV. It infers the structure of the input data to create a baseline schema, which you can further refine using the schema editor.

The interactive SQL editor comes bundled with a collection of stream processing templates that provide baseline SQL code for the most common types of operations such as aggregation, per-event transformation, and filtering. You simply select the template appropriate for your analytics task and then edit the provided code using the SQL editor to customize it for your specific use case.

Amazon Kinesis Data Analytics offers functions optimized for stream processing so that you can easily perform advanced analytics such as anomaly detection and top-K analysis on your streaming data.