Build your own log analytics platform in 20 minutes
The Centralized Logging with OpenSearch (successor to Log Hub) provides comprehensive log management and analysis functions to help you simplify the build of log analytics pipelines and derive business insights. You can use the solution which is built on top of Amazon Opensearch Service to streamline effectively log ingestion, log processing, and log visualization.
Use cases
Security and compliance regulations
Comply with regulatory requirements such as MLPS, GDPR, PCI DSS, and HIPAA. Easily store equipment, network, and application logs in a centralized place for log auditing and threat detection.
Business operations and data analysis
Identify trends and patterns in minutes, and build interactive and intuitive visualization. Derive business insights from logs and empower business decisions with data.
Application and infrastructure troubleshooting
Monitor both application and cloud infrastructure logs with ease, understand and resolve the root cause of issues quickly. Improve observability of your workloads, and achieve better business stability.
Solution overview
The diagram below presents the architecture you can automatically deploy using the solution's implementation guide and accompanying Amazon CloudFormation template.
Centralized Logging with OpenSearch
This solution deploys the Amazon CloudFormation template in your Amazon Web Services Cloud account and completes the following settings.
1.
Amazon CloudFront distributes the frontend web UI assets hosted in
Amazon S3 bucket.
2.
Amazon Cognito user pool or OpenID Connector (OIDC) can be used for authentication.
3.
Amazon AppSync provides the backend GraphQL APIs.
4.
Amazon DynamoDB stores the solution related information as backend database.
5.
Amazon Lambda interacts with other Amazon Services to process core logic of managing log pipelines or log agents, and obtains information updated in DynamoDB tables.
6.
Amazon Step Functions orchestrates on-demand Amazon CloudFormation deployment of a set of predefined stacks for log pipeline management. The log pipeline stacks deploy separate Amazon Web Services resources and are used to collect and process logs and ingest them into
Amazon OpenSearch Service for further analysis and visualization.
7. Service Log Pipelines or Application Log Pipelines are provisioned on demand via Centralized Logging with OpenSearch console.
8.
Amazon Systems Manager and
Amazon EventBridge manage log agents for collecting logs from Application Servers, such as installing log agents (Fluent Bit) for Application servers and monitoring the health status of the agents.
9.
Amazon EC2 or
Amazon EKS installs Fluent Bit agents, and uploads log data to Application Log Pipeline.
10. Application Log Pipelines read, parse, process application logs and ingest them into Amazon OpenSearch Service domains.
11. Service Log Pipelines read, parse, process Amazon service logs and ingest them into Amazon OpenSearch Service domains.
Centralized Logging with OpenSearch
Version 1.0.0
Last updated: 03/2023
Author: Amazon Web Services
Estimated deployment time: 15 min
Benefits
Ease of Use
You can easily deploy the solution into your Amazon Web Services account, and use a web console to ingest both application logs and Amazon service logs. You can facilitate log visualization with out-of-box template dashboards. See implementation guide for details.
Improved Operational Efficiency
The solution combines serverless technologies with built-in high availability, and pay-for-use billing model. The architecture reduces infrastructure management effort, and allows you to focus more on building business use cases.
Open Source and Customization
The solution is open sourced and free for commercial usage. You only pay for the Amazon Web Services usage. If you have different use cases, you can take the source code as reference to make your own implementation.
Explore all Amazon Web Services Solutions
Browse our portfolio of Amazon Web Services-built solutions to common architectural problems.
Find a Partner
Find Amazon Web Services certified consulting and technology partners to help you get started.
