Skip to main content
Sumo Logic

Collect Amazon VPC Flow Logs from CloudWatch using CloudFormation

  1. Last updated
  2. Save as PDF

This page has instructions for collection VPC Flow Logs using a CloudFormation template.  Alternatively, you can Collect Amazon VPC Flow Logs using AWS S3 Source.

This page has instructions for collecting logs for the Amazon VPC Flow Logs app.

Collection process

The diagram below illustrates the collection process for Amazon VPC Flow Logs. VPC is enabled to send logs to Amazon CloudWatch. A Lambda function subscribes to a CloudWatch Log Group to obtain the flow logs, and then sends the data on to a Sumo Logic HTTP Source on a hosted collector. The AWS resources are created by a Sumo-provided CloudFormation template. 

AWSCloudWatch-Collection.png

Step 1: Enable Amazon VPC Flow Logs

You can enable Amazon Virtual Private Cloud (VPC) Flow Logs from the Amazon Web Services (AWS) Management Console, the AWS Command Line Interface (CLI), or by making calls to the Elastic Compute Cloud (EC2) API.

To enable Amazon Virtual Private Cloud (VPC) Flow Logs from the AWS console

  1. Go to VPC management, and go to the VPC list.
  2. Select the VPC.
  3. Click Actions > Create Flow Log.
  4. On the Create Flow Log page, select a Role to use Flow logs.
    1. If you haven't set up IAM permissions, click Set Up Permissions.
      create-vpc-flow-log-cloudwatch.png
    2. From the new tab, VPC Flow Logs is requesting permissions to use resources in your account:
    3. From the IAM Role, select Create a new IAM Role.
    4. Add a Role Name that describes your logs, for example, VPC-Flow-Logs.
    5. Click Allow.
  5. Back in Create Flow Log, enter the new role you created in Role.
  6. In Destination Log Group enter a descriptive name such as VPCFlowLogs.
  7. Click Create Flow Log. It can take up to an hour for the log group to show up in CloudWatch Logs.

Step 2: Configure hosted collector and HTTP source

  1. Configure a Hosted Collector in Sumo Logic.
  2. Configure an HTTP Source in Sumo Logic. When configuring the source:
  3. Under Advanced Options for Logs, for Timestamp Format, click Specify a format.
  4. Format. Enter: 
    epoch
  5. Timestamp locator. Enter:
     \s(\d{10,13})\s\d{10,13} 
  6. Click Save.

Step 3: Create AWS functions and resources 

Follow the steps on Amazon CloudWatch Logs, starting with the Download the CloudFormation template step and ending with the Dealing with alarms step. As you preform the procedure note the additional instructions below, regarding log format and optional environment variables.

Configure LogFormat correctly (Required) 

When you Create a stack on the AWS CloudFormation console, in Step 5, make sure you select either VPC-JSON or VPC-RAW in the LogFormat field in the Specify Details window.

Environment variables for VPC flow log collection (Optional)

When you Configure environment variables for Lambda functions, in addition to the variables listed, you can optionally also define the following environment variables.

Environment variable Description
INCLUDE_SECURITY_GROUP_INFO This option is supported only if you set LogFormat to VPC-JSON

Set to true to include the following fields in logs:

vpc-id
subnet-id
aws-region
security-group-ids
direction

If you set the value to true, follow the instructions in Grant Lambda permissions (Optional).
VPC_CIDR_PREFIX Comma-separated list of IP prefixes for filtering out internal traffic. For example vpcCIDRprefix= 10.8.0.0,10.9.0.0 filters out logs whose destinationIP and sourceIP matches any of the two prefixes 10.8.0.0 and 10.9.0.0. 

Grant Lambda permissions (Optional)

The Lambda function fetches list of Elastic Network Interfaces using the describeNetworkInterfaces API. You need to grant permission to Lambda by adding the following inline policy in the  SumoCWProcessDLQLambda role. See the instructions on Creating Policies on the JSON Tab in AWS help.

Paste the JSON below,  after adding the ARN of the Lambda functions.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "ec2:DescribeNetworkInterfaces",
            "Resource": [


             arn:aws:lambda:<region>:<account_id>:function:<SumoCWProcessDLQLambda-   \                   random_suffix>,


             arn:aws:lambda:<region>:<account_id>:function:<SumoCWLogsLambda-random_suffix>
            ]
        }
    ]
} 

Step 4: Subscribe the Lambda function to the VPC Flow Log group

  1. Select the VPC Flow Log group in the CloudWatch Logs management panel.
    This is the Log Group created in the first part (VPCFlowLogs was used).
  2. Click Actions and select Stream to Lambda Function.
  3. Select the Lambda function created by the CloudFormation template. Its name starts with "SumoCWLogsLambda".
  4. Click Next.
  5. Select JSON for Log Format.
  6. Click Next.
  7. Click Start Streaming. Wait a few minutes, and check to make sure your logs are flowing into Sumo.