Skip to main content
Sumo Logic

Collecting Amazon CloudWatch Logs using a Collector Script

 

For small data volumes, you can use an installed Sumo Logic Collector with a script Source instead of using AWS lambda or Amazon Kinesis to collect Amazon CloudWatch logs.

Create an AWS Access ID and Key pair

Create an AWS user with an AWS Access ID and Key pair. You can follow the instructions in AWS user with an AWS Access ID and Key pair. You can follow the instructions in Grant Access to an AWS S3 Bucket. Specifically, use the instructions from Step 1 to Step 12 to create the user, but at Step 11, use the permission provided below. (To be clear, no S3 bucket permission is required.)

Copy and paste the following code to use as a custom policy:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "1",
"Effect": "Allow",
"Action": [
"logs:DescribeLogGroups",
"logs:DescribeLogStreams",
"logs:DescribeMetricFilters",
"logs:FilterLogEvents",
"logs:GetLogEvents"
],
"Resource": [
"*"
]
}
]
}

 

 

Install a Collector

  1. Install a Collector for your Linux or Windows system. (Make sure you install Collector version 19.119-6 or later.)

You will also configure a Script Source later in this procedure.

Download and Deploy the scripts to the Collector system

The Sumo Logic App for Amazon VPC Flow Logs requires the following files:VPC Flow Logs requires the following files:

To deploy the scripts:

  1. Download the files listed above and deploy them to the same host where the Collector is configured.
  2. Put the package in a folder in the same folder where the Collector is installed, for example,/usr/local/SumoCollector/VPC.
  3. Inside the package, edit the vpc_cwl.properties file to add your AWS Access ID and Key, the region, and the name of the LogGroup used to store the data under a configuration section as shown below. 

 

 

[unique_section_name]
# Predefined value for VPC collection
type = aws_cwl
AccessID = <Your AWS Access ID>
AccessKey = <Your AWS Access Key>
LogGroup = <LogGroupName>
# region, default is us-east-1. Note CWL is supported in: ap-northeast-1, ap-southeast-1, ap-southeast-2, eu-central-1, eu-west-1, us-east-1, us-west-1, and us-west-2
region = us-east-1
# comma separated list of log streams, or don't include if you want to collect from all log streams.
# LogStream = eni-11c6a94a-all,eni-19f34c43-all
# IMPORTANT: file to keep track of last queried timestamp, need a unique file for each section
timestamp = ${path}/timestamp.txt
# start of window to query logs, in epoch milliseconds.
# startTime = 1436377600000
# end of window to query logs, in epoch milliseconds. Use this for a fixed query window, or retrieve archived logs.
# endTime = 1436550400000
# delay time in milliseconds if there is no data
delayDuration = 1000
  1. Save your changes.
  2. Finally, run SumoVPCCollector.sh directly from the CLI to test the settings. If successful, you should see something like this:

Configure a Script Source

  1. Configure a Script Source for the installed Collector to call the main script, as shown. (In this example, we assume the package is located under /usr/local/SumoCollector/VPC. Customize as necessary.)

  • Name. Enter VPC.
  • Source Category. Enter vpc.

 

  • Frequency. Select Every 5 Minutes.
  • Specify a timeout for your command. Activate the check box and select 60 Minutes.
  • Command. Enter /bin/sh.
  • Type a path to the script to execute. Enter the path to the folder where the SumoVPCCollector.sh file is located on the Collector system as you configured in the previous steps.
  • Working Directory. Enter the working directory.

 

  1. Under Advanced, make sure the option Extract timestamp information from log entries is activated.
  2. Make any other configurations necessary, as detailed in Configure a Script Source.
  3. Click Save.