We use machine learning technology to do auto-translation. Click "English" on top navigation bar to check Chinese version.
Monitor Amazon Web Services workloads without a single line of code with Logz.io and Kinesis Firehose
Observability data provides near real-time insights into the health and performance of Amazon Web Services workloads, so that engineers can quickly address production issues and troubleshoot them before widespread customer impact.
As Amazon Web Services workloads grow, observability data has been exploding, which requires flexible big data solutions to handle the throughput of large and unpredictable volumes of observability data.
Solution overview
One option is
Among these observability solutions is Logz.io, which can now ingest metric data from Amazon Kinesis Data Firehose and make it easier to get metrics from your Amazon Web Services account to your Logz.io account for analysis, alerting, and correlation with logs and traces.
In a few clicks and a few configurations, we’ll see how you can start streaming your metric data (and soon, log data!) to Logz.io for storage and analysis.
Prerequisites
- Logz.io account – Create a free trial
here - Logz.io shipping token – Learn about metrics tokens
here . You need to be a Logz.io administrator. - Access to Amazon CloudWatch and Amazon Kinesis Data Firehose with the appropriate permissions to manage HTTP endpoints.
- Appropriate permissions to create an
Amazon Simple Storage Service (Amazon S3) bucket
Sending Amazon CloudWatch metric data to Logz.io with an Amazon Kinesis Data Firehose
Once the integration is configured, Logz.io customers can open the Infrastructure Monitoring product to see their data coming in and populating their dashboards. To see some of the data analytics and correlation you get with Logz.io,
Let’s begin a step-by-step tutorial for setting up the integration.
- Start by going to Amazon Kinesis Data Firehose and creating a delivery stream with Data Firehose.
- Next you select a source and destination. Select Direct Put as the source and Logz.io the destination.
- Next, configure the destination settings. Give the HTTP endpoint a name, which should include logz.io.
- Select from the dropdown the appropriate endpoint you would like to use.
If you’re sending data to a European region, then set it to Logz.io Metrics EU. Or you can use the us-east-1
destination by selecting Logz.io Metrics US .
- Next, add your Logz.io Shipping Token. You can find this by going to Settings in Logz.io and selecting Manage Tokens, which requires Logz.io administrator to access. This ensures that your account is only ingesting data from the defined sources (e.g., this Amazon Kinesis Data Firehose delivery stream).
Keep Content encoding on Disabled and set your desired Retry Duration.
You can also configure Buffer hints to your preferences.
- Next, determine your Backup settings in case something goes wrong. In most cases, it’s only necessary to back up the failed data. Simply choose an Amazon S3 bucket or create a new one to store data if it doesn’t make it to Logz.io. Then, select Create a delivery stream .
Now it’s time to connect Amazon CloudWatch to our Amazon Kinesis Data Firehose Delivery Stream.
- Navigate to Amazon CloudWatch and select Streams in the Metrics menu. Select Create metrics stream .
- Next, you can either select to send all your Amazon CloudWatch metrics to Logz.io, or only metrics from specified namespaces.
In this case, we chose
- Under Configuration, choose the Select an existing Firehose owned by your account option and choose the Amazon Kinesis Data Firehose you just configured.
If you’d like, you can choose additional statistics in the Add additional statistics box, which provides helpful metrics in terms of percentiles to monitor like latency metrics (i.e., which services have the highest average latency). This may increase your costs.
- Lastly, give your metric stream a name and hit Create metric stream .
That’s it! Without writing a single line of code, we configured an integration with Amazon Web Services and Logz.io that enables fast and easy infrastructure monitoring through Amazon CloudWatch data collection.
Your metrics will be stored in Logz.io for 18 months out of the box, without requiring any overhead management.
You can also begin to build dashboards and alerts to begin monitoring – like this Amazon EC2 monitoring dashboard below.
Conclusion
This post demonstrated how to configure an integration with Amazon Web Services and Logz.io for efficient infrastructure monitoring through Amazon CloudWatch.
To learn more about building metrics dashboards in Logz.io, you can watch
Currently, some users might find that they are sending more data than they really need, which can raise costs. In future versions of this integration, it will be easier to narrow down the metrics to reduce costs.
Want to try it yourself?
About the authors
Amos Etzion – Product Manager at Logz.io
Charlie Klein – Product Marketing Manager at Logz.io
Mark Kriaf – Partner Solutions Architect at Amazon Web Services
The mentioned AWS GenAI Services service names relating to generative AI are only available or previewed in the Global Regions. Amazon Web Services China promotes AWS GenAI Services relating to generative AI solely for China-to-global business purposes and/or advanced technology introduction.