Skip to main content
Sumo Logic

Auto-Subscribe AWS Log Groups to a Lambda Function

You can use an Amazon CloudWatch Log Group subscription to access log events from CloudWatch Logs in real time, and send them to Sumo Logic.

You can use an Amazon CloudWatch Log Group subscription to access log events from CloudWatch Logs in real time, and send them to Sumo Logic. 

Sumo’s LogGroup Lambda Connector is a Lambda function that automates the process of creating Amazon CloudWatch Log Group subscriptions.  You can use the connector with Sumo Lambda functions, available at https://github.com/SumoLogic/sumologic-aws-lambda, or with other Lambda functions of your own.

You can deploy this solution in the following ways:

This page provides instructions for both deployment methods, and also covers troubleshooting the connector.

Deploying the SAM application from a serverless repo (recommended method)

This section explains how to deploy the SAM application from a serverless repo. Click links for the related tasks.

To deploy LogGroup Lambda Connector, do the following:
  1. Open a browser window and enter the following URL: https://serverlessrepo.aws.amazon.com/applications

  2. In the Serverless Application Repository, search for sumologic.

  3. Select Show apps that create custom IAM roles or resource policies check box.

  4. Click the sumologic-loggroup-connector,link, and then click Deploy.

  5. In the AWS Lambda > Functions > Application Settings panel, configure the parameters as described in this section.

  6. Scroll to the bottom of the window and click Deploy.

  7. Test the Lambda function.

Deploying the function using a CloudFormation template

This section demonstrates how to deploy the function using a CloudFormation template. 

Step 1: Download the CloudFormation template

Download loggroup-lambda-cft.json, a Sumo-provided CloudFormation template that automates the deployment of the LogGroup Lambda Connector. 

Step 2: Create a stack

In this step, you create a stack using the AWS CloudFormation console.

  1. Log in to the AWS Management Console.
  2. Under Management Tool, select CloudFormation.
  3. Create a new stack by clicking Create Stack.
  4. In Select Template window, choose Upload a template to Amazon S3 and upload loggroup-lambda-cft.json.

    select-template.png
  5. Click Next.
  6. Specify a stack name and click Next. The Specify Details window appears.

    loggroup-connector-details.png
  7. In the Specify Details window, configure parameters as described in this section.
  8. In the Options window, click Next again.
  9. In the Review window, click the checkbox acknowledging that you understand that the template creates IAM resources and click Create. After few seconds CREATE_COMPLETE should appear in the Status column.
  10. Continue with auto-subscribing existing log groups.
  11. Test the Lambda function.

Configuring parameters

This section describes the parameters you can configure for the Lambda function.

  • LambdaARN—Enter the Amazon Resource Name (ARN) of the target Lambda function (the function that will receive CloudWatch logs via the Log Group subscription). To find a function's ARN, open the AWS Lambda console, and select the function from the list. A function's ARN is shown in the upper right corner of the page.

AWS_Lambda-function-parameters.png

  • LogGroupPattern—A Javascript regex to filter Log Groups. Log Groups that match the regex will be subscribed to the connector. Matching is case-insensitive. The placeholder regex Test matches testlogroup, logtestgroup, and LogGroupTest. Replace Test with a  Javascript regex that filters your Log Groups as desired.
  • UseExistingLogs—Controls whether this function will be used to create subscription filters for existing log groups. Select "True" if you want to use the function for subscribing existing log groups.

Using the function to auto-subscribe existing log groups

Follow the steps below to use the connector to subscribe to existing log groups. You selected "True" for the UseExistingLogs option when you created the stack

  1. Disable the CloudWatch Events trigger from the AWS console. Go to  https://aws.amazon.com/lambda/ and click SumoLogGroupLambdaConnector-<unique_string>. Select CloudWatch Events Trigger. Disable the trigger on the CreateLogGroup event, and then click  Save.

loggroup-connector-trigger.png

  1. Modify the USE_EXISTING_LOG_GROUPS environment variable. You can do this while deploying the template and setting the UseExistingLogs parameter to true as described in Step 2: Create a stack. If you have already created the stack, after login to the AWS console,  you can go to the https://aws.amazon.com/lambda/ and click SumoLogGroupLambdaConnector-<unique_string> and set its USE_EXISTING_LOG_GROUPS environment variable to "True".
    loggroup-connector-variables.png
  2.  Invoke the function manually. You can invoke the function using the AWS Management Console or the AWS CLI.
    • To use the console, see Invoke the Lambda Function Manually and Verify Results, Logs, and Metrics in AWS Lambda help.
    • To use the AWS CLI, run the following Lambda CLI invoke command to invoke the function. Note that the command requests asynchronous execution. You can optionally invoke it synchronously by specifying RequestResponse as the invocation-type parameter value.

      aws lambda invoke 
      --invocation-type Event  
      --function-name SumoLogGroupLambdaConnector-<unique_string> 
      --region us-east-2 
      --log-type Tail outputfile.txt

       

Testing the Lambda function

This section demonstrates how to test the Lambda function to ensure that the Connector is functioning properly.

To test the Lambda function, do the following:
  1. Create a Log Group with a name that matches the regex you specified for LogGroupPattern

AWS_LambdaFunction_LoggroupPattern.png

After a few seconds, the Log Group should be subscribed to the Lambda function whose ARN you specified in the LAMBDA_ARN environment variable.

AWS_LambdaFunction_Loggroup-specified.png

  1. View the logs of Lambda function. 
    You  can view the logs generated by SumoLogGroupLambdaConnector-<unique_string> in CloudWatch in the /aws/lambda/SumoLogGroupLambdaConnector-<unique_string> log group.
  2. (Optional) Continue with troubleshooting the connector.

Troubleshooting the connector

This section covers the most common errors you may encounter with a connector and what you can do to resolve the issues.

Permission errors 

If you are getting the permission errors with your Lambda function, you may need to grant CloudWatchLogs permission to invoke your Lambda function.

The following error message indicates that CloudWatch Logs does not have permission to invoke the Lambda function.

{
    "errorMessage": "Could not execute the lambda function. Make sure you have given CloudWatch Logs permission to execute your function.",
    "errorType": "InvalidParameterException",
    "stackTrace": [
        "Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:48:27)",
        "Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:105:20)",
        "Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:77:10)",
        "Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:683:14)",
        "Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)",
        "AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)",
        "/var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10",
        "Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)",
        "Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:685:12)",
        "Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:115:18)"
    ]
}

To grant CloudWatch Logs permission to invoke the Lambda function, run the following AWS CLI command:

aws lambda add-permission --function-name "<function_name>" --statement-id "lambdapermission" --principal "logs.<region>.amazonaws.com" --action "lambda:InvokeFunction" --source-arn "arn:aws:logs:<region>:<account_id>:log-group:*:*" --source-account "<account_id>" --region=<region>

Where:

  • <function_name> is the FunctionName attribute of your target lambda function
  • <region> is the AWS Region where your function is deployed
  • <account> is the AWS Account ID of your aws account

For information about Installing and configuring the AWS CLI, see Installing the AWS Command Line Interface

Log Group belongs to the Lambda function that generated it

The function throws the following exception if the Log Group belongs to the Lambda function that generated it. 

{ "errorMessage": "The log group provided is reserved for the function logs of the 
destination function.", "errorType": "InvalidParameterException", "stackTrace": 
[ 
"Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/json.js:48:27)", 
"Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:105:20)", 
"Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:77:10)", 
"Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:683:14)", 
"Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)", 
"AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)", 
"/var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10", 
"Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)", 
"Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:685:12)", 
"Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:115:18)" 
] }