Skip to main content
Sumo Logic

Amazon CloudWatch Logs

Learn how to collect Amazon CloudWatch Logs.

The preferred method for collecting Amazon CloudWatch Logs is to use the AWS Lambda function provided by Sumo Logic to subscribe to your CloudWatch Log Group. Our AWS Lambda function converts the CloudWatch log format into a format that is compatible with our platform, then POSTs the data directly to a Sumo Logic HTTP Source.  This is the preferred method for the following types of data that are delivered through Amazon CloudWatch Logs:

  1. Custom CloudWatch log data. The AWS Lambda function should handle any log data.  However, you should make sure to test this with your actual data, to ensure that unusually formatted logs are parsed correctly.
  2. Amazon VPC Flow Logs. The AWS Lambda function is compatible with our "Sumo Logic Amazon VPC Flow Logs App." 
  3. AWS Lambda logs. The AWS Lambda function is built for logs generated by your AWS Lambda functions, and is compatible with our "Sumo Logic AWS Lambda App."

The instructions below tell you how to download and configure Sumo Logic's AWS Lambda function for Amazon CloudWatch Logs.

Add a Hosted Collector and HTTP Source

  1. In Sumo Logic, configure a Hosted Collector.
  2. In Sumo Logic, configure an HTTP Source.

Create a Lambda Function

Sumo Logic has created a lambda function for your use with Amazon Web Services (AWS).

https://github.com/SumoLogic/sumologic-aws-lambda/tree/master/cloudwatchlogs/cloudwatchlogs_lambda.js

This file contains a function to collect AWS Lambda logs via CloudWatch Logs. The function extracts and adds a "RequestId" field to each log line to make correlation easier. Download the script file and save it locally.

To add an Amazon lambda function

  1. Sign into the AWS Management Console.
  2. Under Compute, click Lambda.
    Amazon Web Services
  3. Create a new lambda function.

    1. The following introductory page is displayed if the account has no existing lambda functions. Click Get Started Now
    2. Existing lambda functions are displayed if any exist. In this case, click Create a Lambda function.
  4. On the Select blueprint page, click Blank Function
    AWS Select Blueprint
  5. On the Configure triggers page, click Next
    Configure Triggers
  6. On the Configure function page, add a name and other settings for the function and add the Javascript code itself.
    Configure function
    1. Name (Required). Name your lambda function something like "sumo-lambda" or "sumo-vpcflow."
    2. Description (Optional). Provide additional information for future administrators.
    3. Runtime. Select Node.js. (Version 4.3 and 0.10 are both supported.)
    4. Code entry type. Select Edit code inline.
    5. Copy and paste the code from cloudwatchlogs_lambda.js into the text field.
    6. Create the following AWS Lambda environment variables: 
      Environment Variables
      • SUMO_ENDPOINT (Required) - The HTTP Source Address
      • ENCODING (Optional) - The encoding to use when decoding CloudWatch log events. Default is 'utf-8.'
      • SOURCE_CATEGORY_OVERRIDE (Optional) - Override _sourceCategory field in Sumo Logic, or set to "none."
      • SOURCE_HOST_OVERRIDE (Optional) - Override _sourceHost field in Sumo Logic, or set to "none."
      • SOURCE_NAME_OVERRIDE (Optional) - Override _sourceName field in Sumo Logic, or set to "none."
    7. Handler (Required). Use the default, "index.handler," or specify your own. The format is <module name>.<export value>, so the default would call exports.handler in index.js.
    8. Role. The first time only, you will need to set up an Identity and Access Management (IAM) role. If you have no appropriate IAM role defined, select Create new role from template(s). Name the new role and select one or more policy templates.
      If you have an existing role, select Choose an existing role, and select it from the drop down list.
    9. Leave MemoryTimeout, and DLQ Resource as the defaults.
    10. VPC. Select the appropriate VPC flow or No VPC.
    11. KMS key. Specify one of your AWS account's keys, paste in a full KMS key ARN, or use the default, "aws/lambda."
    12. Click Next.
  7. Review the new lambda function configuration.
    Review new lambda function
    Click Create function.

For more information on creating a lambda function, see http://docs.aws.amazon.com/lambda/la...-function.html

Subscribe the Lambda Function

After setting up the Lambda function, next you must map it to an event source. For data stored in CloudWatch logs, such as VPC Flow Logs and Lambda logs, this means subscribing the function to all the CloudWatch log groups that you want to collect. Specifically:

  • VPC Flow Logs. For each Virtual Private Cloud (VPC), the log group is defined by the user when setting up VPC Flow Logs.
  • Lambda Logs. Lambda execution logs are stored on AWS CloudWatch logs. For each Lambda function, AWS creates a log group that uses the following naming convention: /aws/Lambda. For example, for function Foo, the log group name would be /aws/Lambda/Foo.

To subscribe the Lambda Function:

  1. From the AWS CloudWatch Logs console, select the target log group.
  2. Click Action and from the menu, select Stream to AWS Lambda.
  3. Select the function you just created and click Next.
    Choose lambda function
  4. Log Format. Select Amazon VPC Flow Logs or Other.
  5. Leave everything else as shown unchanged, then click Next.
  6. Review the new subscription and click Start Streaming.
  7. Verify that the function appears in the Subscriptions column for the target log group.

Alternate Collection Methods 

Sumo Logic makes alternate methods of collecting logs form AWS CloudWatch available in case the AWS Lambda method is either not available to you, or you have additional requirements. 

  1. Using Amazon Kinesis : If AWS Lambda is not available to you, or you need increased delivery reliability, review how to add Amazon Kinesis to the integration. See Collecting Amazon CloudWatch Logs using Amazon Kinesis.
  2. Using the Sumo Logic Collector and a Script.  If you have a relatively small amount of CloudWatch logs to collect, and you do not want to set up any additional AWS infrastructure, you may install the Sumo Logic Collector agent locally, and run a script that we have developed for CloudWatch logs, with a special focus on Amazon VPC Flow Logs.  See Collect Amazon CloudWatch Logs Using a Collector Script.