Skip to main content
Sumo Logic

Collect Amazon VPC Flow Logs

Enable Amazon VPC Flow Logs

You can enable Amazon Virtual Private Cloud (VPC) Flow Logs from the Amazon Web Services (AWS) Management Console, the AWS Command Line Interface (CLI), or by making calls to the Elastic Compute Cloud (EC2) API.

To enable Amazon Flow Logs for your VPC, refer to the Amazon documentation

Collect Amazon VPC Flow Logs via CloudWatch

All Amazon VPC Flow Logs are delivered via the Amazon CloudWatch Logs service. Sumo Logic provides the following mechanisms for the collection of these events:

  • AWS Lambda functions. Sumo Logic has an AWS Lambda function for CloudWatch logs that is built specifically for  VPC flow logs and is compatible with the Sumo Logic Amazon VPC Flow Logs App.  See AWS Lambda.
  • Amazon Kinesis. If AWS Lambda is not available to you, or you need increased delivery reliability, you can add Amazon Kinesis to the integration. See Collection Amazon CloudWatch Logs Using Amazon Kinesis.
  • Using the Sumo Logic Collector and a Script. If you have a relatively small amount of CloudWatch logs to collect, and you do not want to set up any additional AWS infrastructure, you can install the Sumo Logic Collector agent locally and run a Sumo Logic script designed for Amazon VPC Flow Logs. See Collect Amazon CloudWatch Logs Using a Collector Script.

Create a Lambda Function

Sumo Logic has created a lambda function for your use with Amazon Web Services (AWS).

https://github.com/SumoLogic/sumologic-aws-lambda/tree/master/cloudwatchlogs/cloudwatchlogs_lambda.js

This file contains a function to collect AWS Lambda logs via CloudWatch Logs. The function extracts and adds a "RequestId" field to each log line to make correlation easier. Download the script file and save it locally.

To add an Amazon lambda function

  1. Sign into the AWS Management Console.
  2. Under Compute, click Lambda.
    Amazon Web Services
  3. Create a new lambda function.

    1. The following introductory page is displayed if the account has no existing lambda functions. Click Get Started Now
    2. Existing lambda functions are displayed if any exist. In this case, click Create a Lambda function.
  4. On the Select blueprint page, click Blank Function
    AWS Select Blueprint
  5. On the Configure triggers page, click Next
    Configure Triggers
  6. On the Configure function page, add a name and other settings for the function and add the Javascript code itself.
    Configure function
    1. Name (Required). Name your lambda function something like "sumo-lambda" or "sumo-vpcflow."
    2. Description (Optional). Provide additional information for future administrators.
    3. Runtime. Select Node.js. (Version 4.3 and 0.10 are both supported.)
    4. Code entry type. Select Edit code inline.
    5. Copy and paste the code from cloudwatchlogs_lambda.js into the text field.
    6. Create the following AWS Lambda environment variables: 
      Environment Variables
      • SUMO_ENDPOINT (Required) - The HTTP Source Address
      • ENCODING (Optional) - The encoding to use when decoding CloudWatch log events. Default is 'utf-8.'
      • SOURCE_CATEGORY_OVERRIDE (Optional) - Override _sourceCategory field in Sumo Logic, or set to "none."
      • SOURCE_HOST_OVERRIDE (Optional) - Override _sourceHost field in Sumo Logic, or set to "none."
      • SOURCE_NAME_OVERRIDE (Optional) - Override _sourceName field in Sumo Logic, or set to "none."
    7. Handler (Required). Use the default, "index.handler," or specify your own. The format is <module name>.<export value>, so the default would call exports.handler in index.js.
    8. Role. The first time only, you will need to set up an Identity and Access Management (IAM) role. If you have no appropriate IAM role defined, select Create new role from template(s). Name the new role and select one or more policy templates.
      If you have an existing role, select Choose an existing role, and select it from the drop down list.
    9. Leave MemoryTimeout, and DLQ Resource as the defaults.
    10. VPC. Select the appropriate VPC flow or No VPC.
    11. KMS key. Specify one of your AWS account's keys, paste in a full KMS key ARN, or use the default, "aws/lambda."
    12. Click Next.
  7. Review the new lambda function configuration.
    Review new lambda function
    Click Create function.

For more information on creating a lambda function, see http://docs.aws.amazon.com/lambda/la...-function.html

Subscribe the Lambda Function

After setting up the Lambda function, next you must map it to an event source. For data stored in CloudWatch logs, such as VPC Flow Logs and Lambda logs, this means subscribing the function to all the CloudWatch log groups that you want to collect. Specifically:

  • VPC Flow Logs. For each Virtual Private Cloud (VPC), the log group is defined by the user when setting up VPC Flow Logs.
  • Lambda Logs. Lambda execution logs are stored on AWS CloudWatch logs. For each Lambda function, AWS creates a log group that uses the following naming convention: /aws/Lambda. For example, for function Foo, the log group name would be /aws/Lambda/Foo.

To subscribe the Lambda Function:

  1. From the AWS CloudWatch Logs console, select the target log group.
  2. Click Action and from the menu, select Stream to AWS Lambda.
  3. Select the function you just created and click Next.
    Choose lambda function
  4. Log Format. Select Amazon VPC Flow Logs or Other.
  5. Leave everything else as shown unchanged, then click Next.
  6. Review the new subscription and click Start Streaming.
  7. Verify that the function appears in the Subscriptions column for the target log group.