Skip to main content

Centralized Logging with OpenSearch (Successor to Log Hub)

Build your own log analytics platform in 20 minutes

View implementation guide

What does this solution do?

The Centralized Logging with OpenSearch (successor to Log Hub) provides comprehensive log management and analysis functions to help you simplify the build of log analytics pipelines and derive business insights. You can use the solution which is built on top of Amazon Opensearch Service to streamline effectively log ingestion, log processing, and log visualization. 

Use cases

Security and compliance regulations

Comply with regulatory requirements such as MLPS, GDPR, PCI DSS, and HIPAA. Easily store equipment, network, and application logs in a centralized place for log auditing and threat detection.

Business operations and data analysis

Identify trends and patterns in minutes, and build interactive and intuitive visualization. Derive business insights from logs and empower business decisions with data. 

Application and infrastructure troubleshooting

Monitor both application and cloud infrastructure logs with ease, understand and resolve the root cause of issues quickly. Improve observability of your workloads, and achieve better business stability. 

Solution overview

The diagram below presents the architecture you can automatically deploy using the solution's implementation guide and accompanying Amazon CloudFormation template.

solution
opensearch-service
central-diag

Centralized Logging with OpenSearch

This solution deploys the Amazon CloudFormation template in your Amazon Web Services Cloud account and completes the following settings.

1.  Amazon CloudFront distributes the frontend web UI assets hosted in  Amazon S3 bucket.

2.  Amazon Cognito user pool or OpenID Connector (OIDC) can be used for authentication.

3.  Amazon AppSync provides the backend GraphQL APIs.

4. Amazon DynamoDB stores the solution related information as backend database.

5. Amazon Lambda interacts with other Amazon Services to process core logic of managing log pipelines or log agents, and obtains information updated in DynamoDB tables.

6. Amazon Step Functions orchestrates on-demand Amazon CloudFormation deployment of a set of predefined stacks for log pipeline management. The log pipeline stacks deploy separate Amazon Web Services resources and are used to collect and process logs and ingest them into  Amazon OpenSearch Service for further analysis and visualization.

7. Service Log Pipelines or Application Log Pipelines are provisioned on demand via Centralized Logging with OpenSearch console.

8.  Amazon Systems Manager and  Amazon EventBridge manage log agents for collecting logs from Application Servers, such as installing log agents (Fluent Bit) for Application servers and monitoring the health status of the agents.

9. Amazon EC2 or  Amazon EKS installs Fluent Bit agents, and uploads log data to Application Log Pipeline.

10. Application Log Pipelines read, parse, process application logs and ingest them into Amazon OpenSearch Service domains or Light Engine.

11. Service Log Pipelines read, parse, process Amazon service logs and ingest them into Amazon OpenSearch Service domains or Light Engine.

Benefits

Ease of Use

You can easily deploy the solution into your Amazon Web Services account, and use a web console to ingest both application logs and Amazon service logs. You can facilitate log visualization with out-of-box template dashboards. See implementation guide for details. 

Improved Operational Efficiency

The solution combines serverless technologies with built-in high availability, and pay-for-use billing model. The architecture reduces infrastructure management effort, and allows you to focus more on building business use cases.

Open Source and Customization

The solution is open sourced and free for commercial usage. You only pay for the Amazon Web Services usage. If you have different use cases, you can take the source code as reference to make your own implementation.