Collect Amazon VPC Flow Logs from CloudWatch using CloudFormation
This page has instructions for collecting VPC Flow Logs using a CloudFormation template. Alternatively, you can Collect Amazon VPC Flow Logs using AWS S3 Source.
This page has instructions for collecting logs for the Amazon VPC Flow Logs app.
Collection process
The diagram below illustrates the collection process for Amazon VPC Flow Logs. VPC is enabled to send logs to Amazon CloudWatch. A Lambda function subscribes to a CloudWatch Log Group to obtain the flow logs, and then sends the data on to a Sumo Logic HTTP Source on a hosted collector. The AWS resources are created by a Sumo-provided CloudFormation template.
Step 1: Enable Amazon VPC Flow Logs
You can enable Amazon Virtual Private Cloud (VPC) Flow Logs from the Amazon Web Services (AWS) Management Console, the AWS Command Line Interface (CLI), or by making calls to the Elastic Compute Cloud (EC2) API.
To enable Amazon Virtual Private Cloud (VPC) Flow Logs from the AWS console
- Go to VPC management, and go to the VPC list.
- Select the VPC.
- Click Actions > Create Flow Log.
- On the Create Flow Log page, select a Role to use Flow logs.
- If you haven't set up IAM permissions, click Set Up Permissions.
- From the new tab, VPC Flow Logs is requesting permissions to use resources in your account:
- From the IAM Role, select Create a new IAM Role.
- Add a Role Name that describes your logs, for example, VPC-Flow-Logs.
- Click Allow.
- If you haven't set up IAM permissions, click Set Up Permissions.
- Back in Create Flow Log, enter the new role you created in Role.
- In Destination Log Group enter a descriptive name such as VPCFlowLogs.
- Click Create Flow Log. It can take up to an hour for the log group to show up in CloudWatch Logs.
Step 2: Configure hosted collector and HTTP source
- Configure a Hosted Collector in Sumo Logic.
- Configure an HTTP Source in Sumo Logic. When configuring the source:
- Under Advanced Options for Logs, for Timestamp Format, click Specify a format.
- Format. Enter:
epoch
- Timestamp locator. Enter:
\s(\d{10,13})\s\d{10,13}
- Click Save.
Step 3: Create AWS functions and resources
Follow the steps on Amazon CloudWatch Logs, starting with the Download the CloudFormation template step and ending with the Dealing with alarms step. As you perform the procedure note the additional instructions below, regarding log format and optional environment variables.
Configure LogFormat correctly (Required)
When you Create a stack on the AWS CloudFormation console, in Step 5, make sure you select either VPC-JSON or VPC-RAW in the LogFormat field in the Specify Details window.
Environment variables for VPC flow log collection (Optional)
When you Configure environment variables for Lambda functions, in addition to the variables listed, you can optionally also define the following environment variables.
Environment variable | Description |
INCLUDE_SECURITY_GROUP_INFO |
This option is supported only if you set LogFormat to VPC-JSONSet to true to include the following fields in logs:vpc-id subnet-id aws-region security-group-ids direction If you set the value to true , follow the instructions in Grant Lambda permissions (Optional). |
VPC_CIDR_PREFIX |
Comma-separated list of IP prefixes for filtering out internal traffic. For example
|
Grant Lambda permissions (Optional)
The Lambda function fetches list of Elastic Network Interfaces using the describeNetworkInterfaces
API. You need to grant permission to Lambda by adding the following inline policy in the SumoCWLambdaExecutionRole
role. See the instructions on Creating Policies on the JSON Tab in AWS help.
Paste the JSON below, after adding the ARN of the Lambda functions.
{ "Version": "2012-10-17", "Statement": [ { "Sid": "DescribeENILambdaPerms", "Effect": "Allow", "Action": "ec2:DescribeNetworkInterfaces", "Resource": "*" } ] }
Step 4: Subscribe the Lambda function to the VPC Flow Log group
- Select the VPC Flow Log group in the CloudWatch Logs management panel.
This is the Log Group created in the first part (VPCFlowLogs was used). - Click Actions and select Stream to Lambda Function.
- Select the Lambda function created by the CloudFormation template. Its name starts with "SumoCWLogsLambda".
- Click Next.
- Select JSON for Log Format.
- Click Next.
- Click Start Streaming. Wait a few minutes, and check to make sure your logs are flowing into Sumo.