Optimize AWS event and log collection using common design patterns

If your organization operates within hundreds or thousands of AWS accounts, most likely you’ve been searching for better ways to optimize how you collect and process events and logs from your AWS services. Event and log information is a key component when gaining valuable business insights and providing capabilities like cloud security posture management (CSPM), cloud-native application protection platform (CNAPP), security Information and event management (SIEM), extended detection and response (XDR), and more.

Many AWS customers follow a multi-account strategy to establish their cloud foundation, as described in the topic Organizing Your AWS Environment Using Multiple Accounts. Using this strategy, they can enable multiple AWS services at the AWS Organizations level and then transfer events and logs from these services to a dedicated AWS account. Oftentimes, they use open source, in-house, or AWS Partner tools to visualize the collected data for analysis.

In our experience working with customers, we’ve learned that a top priority for organizations is to collect event and log data in the most efficient and cost-effective way. In this blog post, we discuss some common approaches for collecting data from multiple AWS services across your organization. We also present some common patterns that you can reuse to consume the data. Code snippets are available to help you build similar solutions in your own environment.

Overview

This blog post describes the following approaches and patterns for collecting and consuming event and log data.

• Approach 1: Collecting logs in a centralized Amazon CloudWatch monitoring account
• Approach 2: Collecting logs in an Amazon S3 bucket in a centralized log archive account
  • Pattern A: Sending Amazon S3 event notifications through Amazon SQS
  • Pattern B: Sending Amazon S3 event notifications through Amazon SNS
  • Pattern C: Fanning out Amazon S3 event notifications using a combination of Amazon SNS and Amazon SQS
  • Pattern D: Performing centralized logging with Amazon OpenSearch

Prerequisites

You must have a multi-account environment with centralized log collection that is configured to a log archive account. For more information, see What is AWS Control Tower?

Confirm that the following AWS services adhere to security best practices (not covered in this blog post). For more information, see:

Approach 1: Collecting logs in a centralized Amazon CloudWatch monitoring account

As shown in the following diagram, logs from multiple AWS services and accounts are collected in cross-account CloudWatch log groups within a designated AWS monitoring account.

Note: CloudWatch log collection requires separate configuration for each AWS Region.

Approach 2: Collecting logs in an Amazon S3 bucket in a centralized log archive account

As shown in the following diagram, logs and events from multiple AWS services (for example, AWS Config, AWS CloudTrail, Amazon GuardDuty, VPC Flow Logs, and more) within an AWS organization are collected in an Amazon S3 bucket located in a designated AWS log archive account. The mechanism for collecting events and logs varies by AWS service. For details, see the AWS Documentation website, and navigate to each service area.

Note: If using AWS Control Tower, use the centralized log account to collect and store logs. This log account is assigned when you create the landing zone.

When logs are collected from multiple AWS services and accounts into a centralized Amazon S3 bucket, you can use Amazon S3 event notifications to send the logs to partner tools and perform other processing tasks. In the following sections, we describe some common patterns that you can use to configure these processes.

Pattern A: Sending Amazon S3 event notifications through Amazon SQS

When logs are added to a centralized S3 bucket, event notifications are generated and added to an Amazon SQS queue, which is located in the AWS account that contains the S3 bucket. Then, the events are consumed and processed by partner applications.

The following diagram shows the overall flow of this pattern.

1. When a new data object is created, Amazon S3 event notifications are sent to an Amazon SQS queue from a centralized log location in the log archive account.
2. The partner engine polls the Amazon SQS queue for new messages. Once the partner engine receives messages, it processes the objects using a cross-account IAM role that’s configured in the log archive account.
3. The partner engine gets messages and the details of the newly created objects from the Amazon SQS queue.
4. The partner engine gets the data objects from the S3 bucket so they can be processed.
5. After the partner engine processes the messages, it alerts Amazon SQS to delete the messages from the queue.

Implementing this pattern is a two-part process:

• Part 1: You configure your environment to add message events to the Amazon SQS queue when a new object is created. To try this implementation, deploy this CloudFormation template to a centralized logging account and configure Amazon S3 event notifications to send events to your Amazon SQS queue (created in the same account as the S3 bucket) when a new object is created.
• Part 2: You define the way the logs are collected and consumed via your partner engine. To try this implementation, deploy this CloudFormation template in your consumer account. Your partner engine receives messages from the Amazon SQS queue (located in the log archive account). A cross-account IAM role allows your partner engine to read new objects from the S3 bucket.

The partner engine processes the Amazon S3 events and then loads the object data into your data sources. This engine can be hosted locally within your environment for open source and custom-built software, or it can be hosted on partner-owned AWS accounts for SaaS products (most common).

The implementation and architecture of the partner engine can vary. For example, the engine can be serverless or event-driven and can use multiple AWS services and resources such as AWS Lambda, Amazon Elastic Compute Cloud (Amazon EC2), or services with containers.

Pros:

• You can use Amazon SQS to batch-process multiple Amazon S3 events before invoking a Lambda function. This improves performance by reducing cold starts and function invocations.
• Amazon SQS provides better visibility of the backlog of unprocessed or failed events.
• Amazon SQS provides at-least-once processing of messages for standard queues or exactly-once processing when using FIFO (First-In-First-Out) queues.

Cons:

• Additional configuration is needed for Amazon SQS to fan out notifications to multiple subscribers simultaneously.
• Amazon SQS incurs fees based on the number of requests and amount of data processed.

Pattern B: Sending Amazon S3 event notifications through Amazon SNS

As shown in the following diagram, the architecture of this pattern is almost identical to the previous pattern, except in this case, the events are sent to an Amazon SNS topic rather than processed via an Amazon SQS queue.

1. When new data objects are created, Amazon S3 event notifications are sent to an Amazon SNS topic from a centralized log location in the log archive account.
2. When subscribed to the event, the partner engine processes the newly created objects using a cross-account IAM role that’s configured in the log archive account. During this processing stage, the partner engine receives the data objects from the S3 bucket and the details of those objects from the Amazon SNS topic.

Pros:

• Amazon SNS can fan out notifications to a large number of subscribers simultaneously with high throughput.
• Amazon SNS doesn’t require processing fees.

Con: SNS provides at-least-once delivery to subscribers but cannot guarantee that subscribers receive messages when client- and/or server-side errors occur.

To try this pattern, deploy this CloudFormation template in the centralized logging account and configure an Amazon S3 event notification to send messages to an Amazon SNS topic (in the same account) when a new object is created in the S3 bucket.

Pattern C: Fanning out Amazon S3 event notifications using a combination of Amazon SNS and Amazon SQS

When logs are added to a centralized S3 bucket, event notifications are generated and added to an Amazon SQS queue, which is located in the AWS account that contains the S3 bucket. Next, the events are sent to an Amazon SNS topic through an Amazon EventBridge pipe. This allows multiple engines to consume the logs.

Note: The dynamics involved in consuming these events and processing them is beyond the scope of this blog.

1. Amazon S3 event notification messages are added to an Amazon SQS queue from a centralized log location in the log archive account.
2. Messages are routed from the Amazon SQS queue to an EventBridge pipe.
3. The Amazon SNS topic receives the messages from the EventBridge pipe.
4. The Amazon SNS topic fans out the messages to multiple subscribers, for example via queues that serve partner applications or a Lambda function configured to process the messages.
5. Messages are consumed by subscribers who are either part of a customer-owned tooling account or partner-hosted application environment. You can extend this step with additional logic for consuming and using logs with applications to gain additional insights.

To try this pattern, deploy this CloudFormation template in the centralized logging account and configure an Amazon S3 event notification to send messages to an Amazon SQS queue when a new object is created, and then to an Amazon SNS topic to fan out the message to multiple subscribers. Subscribers can be third-party partner tools or other partner-owned AWS accounts.

Pro: This approach provides better fan-out capability if you need to consume the logs through multiple tools.

Con: Using additional AWS services like EventBridge incurs more cost.

Pattern D: Performing centralized logging with Amazon OpenSearch

You can use OpenSearch to perform centralized logging for multiple AWS services and features such as AWS CloudTrail, VPC Flow Logs, and others. You can also use OpenSearch to log and monitor your AWS applications. For details, see Centralized Logging with OpenSearch.

Cleanup

If you don’t use the solution, or you decide to uninstall it, delete the CloudFormation stack to avoid incurring additional costs. For details, see Deleting a stack on the AWS CloudFormation console.

Conclusion

In this blog post, we introduced you to some common approaches for collecting logs from multiple AWS services in a centralized location, allowing you to unlock valuable insights. You also saw some common design patterns for consuming logs and learned about the pros and cons associated with each approach.

We encourage you to use the provided sample CloudFormation templates as a reference when planning how to channel your centralized logs to third-party or open-source tools. If you have comments or feedback about this post, please submit them in the discussion area of this page.

About the authors