Skip to main content
Sumo Logic

Auto-Subscribe AWS Log Groups to a Lambda Function

You can use an Amazon CloudWatch Log Group subscription to access log events from CloudWatch Logs in real time, and send them to Sumo Logic.

You can use an Amazon CloudWatch Log Group subscription to access log events from CloudWatch Logs in real time, and send them to Sumo Logic. 

Sumo’s LogGroup Lambda Connector is a Lambda function that automates the process of creating Amazon CloudWatch Log Group subscriptions.  

You can use the connector with Sumo Lambda functions, available at https://github.com/SumoLogic/sumologic-aws-lambda, or with other Lambda functions of your own.

The connector can be used with existing or new log groups.

This page has instructions for configuring and deploying the LogGroup Lambda Connector using a CloudFormation template.

Step 1. Download the CloudFormation template

Download loggroup-lambda-cft.json, a Sumo-provided CloudFormation template that automates the deployment of the LogGroup Lambda Connector. The template creates the following resources:

Resource Description
PermissionForEventsToInvokeLambda Permission to CloudTrail events for invoking the Lambda function (SumoLogGroupLambdaConnector).
SumoLGCnCreateLogGroupTrigger A CloudTrail Event Rule which triggers the Lambda function (SumoLogGroupLambdaConnector) on the CreateLogGroup event.
SumoLGCnLambdaExecutionRole An IAM Role for the Lambda function that defines permissions to create subscription filters and CloudWatch logs.
SumoLogGroupLambdaConnector The Lambda function responsible for creating a subscription filter on Log Groups that match specified filter criteria.

Step 2: Create a stack

In this step, you create a stack using the AWS CloudFormation console.

  1. Log in to the AWS Management Console.
  2. Under Management Tool, select CloudFormation.
  3. Create a new stack by clicking Create Stack.
  4. In Select Template window, choose Upload a template to Amazon S3 and upload loggroup-lambda-cft.json.
    select-template.png
  5. Click Next.
  6. Specify a stack name and click Next. The Specify Details window appears.

    loggroup-connector-details.png
  7. In the Specify Details window, define the following parameters:
    1. LambdaARN. Enter the Amazon Resource Name (ARN) of the target Lambda function (the function that will receive CloudWatch logs via the Log Group subscription). To find a function's ARN, open the AWS Lambda console, and select the function from the list. A function's ARN is shown in the upper right corner of the page.
      arn.png
    2. LogGroupPattern. A Javascript regex to filter Log Groups. Log Groups that match the regex will be subscribed to the connector. Matching is case-insensitive. The placeholder regex Test  matches testlogroup, logtestgroup, and LogGroupTest. Replace Test with a  Javascript regex that filters your Log Groups as desired. 
    3. UseExistingLogs. Controls whether this function will be used to create subscription filters for existing log groups. Select "True" if you want to use the function for subscribing existing log groups.
  8. In the Options window, click Next again.
  9. In the Review window, click the checkbox acknowledging that you understand that the template creates IAM resources and click Create.
  10. After few seconds CREATE_COMPLETE should appear in the Status column.

Step 3: Giving Invoke Lambda Permission to CloudWatch Logs (Optional)

This step is not necessary If you are using Sumo Logic Lambda functions.

If you are using another Lambda function, you may need to grant CloudWatchLogs permission to  invoke your Lambda function. Test your function. If permission error occurs, see Permission errors below.

Step 4: Test the Lambda function

To test the Lambda function

  1. Create a Log Group with a name that matches the regex you specified for LogGroupPattern.
    test1.png
  2. After a few seconds, the Log Group should be subscribed to the Lambda function whose ARN you specified in the LAMBDA_ARN environment variable.
    test2.png

Step 5. Use the function to auto-subscribe existing log groups

Follow the steps below to use the connector to subscribe to existing log groups. (You selected "True" for the UseExistingLogs option when you created the stack in the previous step.) 

  1. Disable the CloudWatch Events trigger from the AWS console. Go to  https://aws.amazon.com/lambda/ and click SumoLogGroupLambdaConnector-<unique_string>. Select CloudWatch Events Trigger. Disable the trigger on the CreateLogGroup event. Click  Save.
    loggroup-connector-trigger.png
  2. Modify the USE_EXISTING_LOG_GROUPS environment variable. You can do this while deploying the template and setting the UseExistingLogs parameter to true as described in Step 2: Create a stack. If you have already created the stack, after login to the AWS console,  you can go to the https://aws.amazon.com/lambda/ and click SumoLogGroupLambdaConnector-<unique_string> and set its USE_EXISTING_LOG_GROUPS environment variable to "True".
    loggroup-connector-variables.png
  3.  Invoke the function manually. You can invoke the function using the AWS Management Console or the AWS CLI.
    • To use the console, see Invoke the Lambda Function Manually and Verify Results, Logs, and Metrics in AWS Lambda help.
    • To use the AWS CLI, run the following Lambda CLI invoke command to invoke the function. Note that the command requests asynchronous execution. You can optionally invoke it synchronously by specifying RequestResponse as the invocation-type parameter value.

      aws lambda invoke 
      --invocation-type Event  
      --function-name SumoLogGroupLambdaConnector-<unique_string> 
      --region us-east-2 
      --log-type Tail outputfile.txt


      For information about installing the CLI, see Installing the AWS Command Line Interface.

Troubleshooting the connector

You  can view the logs generated by SumoLogGroupLambdaConnector-<unique_string> in CloudWatch in the /aws/lambda/SumoLogGroupLambdaConnector-<unique_string> log group.  

Permission errors 

The error message below indicates that CloudWatch Logs does not have permission to invoke the Lambda function.

{
    "errorMessage": "Could not execute the lambda function. Make sure you have given CloudWatch Logs permission to execute your function.",
    "errorType": "InvalidParameterException",
    "stackTrace": [
        "Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:48:27)",
        "Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:105:20)",
        "Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:77:10)",
        "Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:683:14)",
        "Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)",
        "AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)",
        "/var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10",
        "Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)",
        "Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:685:12)",
        "Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:115:18)"
    ]
}

To grant CloudWatch Logs permission to invoke the Lambda function, run the following AWS CLI command:

aws lambda add-permission --function-name "<function_name>" --statement-id "lambdapermission" --principal "logs.<region>.amazonaws.com" --action "lambda:InvokeFunction" --source-arn "arn:aws:logs:<region>:<account_id>:log-group:*:*" --source-account "<account_id>" --region=<region>

Where:

  • <function_name> is the FunctionName attribute of your target lambda function
  • <region> is the AWS Region where your function is deployed
  • <account> is the AWS Account ID of your aws account

For information about Installing and configuring the AWS CLI, see Installing the AWS Command Line Interface

Log Group belongs to the Lambda function that generated it

The function throws the following exception if the Log Group belongs to the Lambda function that generated it. 

{ "errorMessage": "The log group provided is reserved for the function logs of the destination function.", "errorType": "InvalidParameterException", "stackTrace": [ "Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:48:27)", "Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:105:20)", "Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:77:10)", "Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:683:14)", "Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)", "AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)", "/var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10", "Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)", "Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:685:12)", "Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:115:18)" ] }