Build your own log analytics platform in 20 minutes

The Log Hub solution provides comprehensive log management and analysis functions to help you simplify the build of log analytics pipelines and derive business insights. You can use the solution which is built on top of Amazon Opensearch Service to streamline effectively log ingestion, log processing, and log visualization. 

Use cases

Security and compliance regulations

Comply with regulatory requirements such as MLPS, GDPR, PCI DSS, and HIPAA. Easily store equipment, network, and application logs in a centralized place for log auditing and threat detection.

Business operations and data analysis

Identify trends and patterns in minutes, and build interactive and intuitive visualization. Derive business insights from logs and empower business decisions with data.

Application and infrastructure troubleshooting

Monitor both application and cloud infrastructure logs with ease, understand and resolve the root cause of issues quickly. Improve observability of your workloads, and achieve better business stability.

Solution overview

The diagram below presents the architecture you can automatically deploy using the solution's implementation guide and accompanying Amazon CloudFormation template.

Log Hub

This solution deploys the Amazon CloudFormation template in your Amazon Web Services Cloud account and completes the following settings.
 
1.   Amazon CloudFront distributes the frontend web UI assets hosted in  Amazon S3 bucket.
 
2. For Amazon Web Services Global Regions,  Amazon Cognito user pool provides authentication for backend. For Amazon Web Services China Regions, OpenID Connector (OIDC) is used.
 
3.  Amazon AppSync provides the backend GraphQL APIs.
 
4. Amazon DynamoDB stores the solution related information as backend database.
 
5. Amazon Lambda interacts with other Amazon Services to process core logic of managing log pipelines or log agents, and obtains information updated in DynamoDB tables.
 
6. Amazon Step Functions orchestrates on-demand Amazon CloudFormation deployment of a set of predefined stacks for log pipeline management. The log pipeline stacks deploy separate Amazon Web Services resources and are used to collect and process logs and ingest them into  Amazon OpenSearch Service for further analysis and visualization.
 
7.  The solution provisions Service Log Pipeline or Application Log Pipeline on demand.
 
8.  Amazon Systems Manager and  Amazon EventBridge manage log agents for collecting logs from Application Servers, such as installing log agents (Fluent Bit) for Application servers and monitoring the health status of the agents.
 
9. Amazon EC2 or  Amazon EKS installs Fluent Bit agents, and uploads log data to Application Log Pipeline.
 
10. Application Log Pipelines read, parse, process application logs and ingest them into Amazon OpenSearch.
 
11. Service Log Pipelines read, parse, process Amazon service logs and ingest them into Amazon OpenSearch.
 
 

Benefits

Ease of use

You can easily deploy the solution into your Amazon Web Services account, and use a web console to ingest both application logs and Amazon service logs. You can facilitate log visualization with out-of-box template dashboards. See implementation guide for details.

Improved Operational Efficiency

The solution combines serverless technologies with built-in high availability, and pay-for-use billing model. The architecture reduces infrastructure management effort, and allows you to focus more on building business use cases.

Open Source and Customization

The solution is open sourced and free for commercial usage. You only pay for the Amazon Web Services usage. If you have different use cases, you can take the source code as reference to make your own implementation.
Explore all Amazon Web Services Solutions

Browse our portfolio of Amazon Web Services-built solutions to common architectural problems.

Learn more 
Find a Partner

Find Amazon Web Services certified consulting and technology partners to help you get started.

Learn more 
Start building in the console

Sign-up and start exploring our services.

Get started