Collect Logs and Metrics for the Amazon Kinesis - Streams App

Collect Metrics for Amazon Kinesis - Streams 

1. Configure a Hosted Collector.
2. Configure an Amazon CloudWatch Source for Metrics.
   1. Name. Enter a name to display for the new Source.
   2. Description. Enter an optional description.
   3. Regions. Select your Amazon Regions for Kinesis.
   4. Namespaces. Select AWS/Kinesis.
   5. Source Category. Enter a source category. For example, kinesis_metrics.
   6. Access Key ID and Secret Access Key. Enter your Amazon Access Key ID and Secret Access Key.
   7. Scan Interval. Use the default of 5 minutes, or enter the frequency Sumo Logic will scan your CloudWatch Sources for new data.
3. Click Save.

Collect Amazon Kinesis - Streams Events using CloudTrail

1. To your Hosted Collector, add an AWS CloudTrail Source.
   1. Name. Enter a name to display for the new Source.
   2. Description. Enter an optional description.
   3. S3 Region. Select the Amazon Region for your Kinesis S3 bucket.
   4. Bucket Name. Enter the exact name of your Kinesis S3 bucket.
   5. Path Expression. Enter the string that matches the S3 objects you'd like to collect. You can use a wildcard (*) in this string. (DO NOT use a leading forward slash. See Amazon Path Expressions.)
   6. Source Category. Enter a source category. For example, kinesis_event.
   7. Access Key ID and Secret Access Key. Enter your Amazon Access Key ID and Secret Access Key.
   8. Scan Interval. Use the default of 5 minutes. Alternately, enter the frequency Sumo Logic will scan your S3 bucket for new data.
   9. Enable Timestamp Parsing. Select the check box.
   10. Time Zone. Select Ignore time zone from log file and instead use, and select UTC.
   11. Timestamp Format. Select Automatically detect the format.
   12. Enable Multiline Processing. Select the check box, and select Infer Boundaries.
2. Click Save.

Sample Log Message

{  
   "eventVersion":"1.01",
   "userIdentity":{  
      "type":"IAMUser",
      "principalId":"EX_PRINCIPAL_ID",
      "arn":"arn:aws:iam::012345678910:user/Alice",
      "accountId":"012345678910",
      "accessKeyId":"vmLwWTxSQrcvzD",
      "userName":"Gosia"
   },
   "eventTime":"2017-11-01T21:23:30+0000",
   "errorCode":"LimitExceedException",
   "errorMessage":"Rate exceeded for stream CWL-Kinesis under account 656757657843",
   "eventSource":"kinesis.amazonaws.com",
   "eventName":"MergeShards",
   "awsRegion":"us-east-2 ",
   "sourceIPAddress":"187.185.157.125",
   "userAgent":"aws-sdk-java/unknown-version Linux/x.xx",
   "requestParameters":{  
      "streamName":"GoodStream",
      "adjacentShardToMerge":"shardId-000000000002",
      "shardToMerge":"shardId-000000000001"
   },
   "responseElements":null,
   "requestID":"e9f9c8eb-c757-11e3-bf1d-6948db3cd570",
   "eventID":"77cf0d06-ce90-42da-9576-71986fec411f"
}

Query Sample

Details of errors in events

_sourceCategory=aws/kinesis* "kinesis.amazonaws.com" errorCode
| json field=_raw "eventSource", "eventName", "awsRegion", "sourceIPAddress","userAgent" nodrop
| json field=_raw "requestParameters.streamName" as streamName nodrop
| json field=_raw "userIdentity.sessionContext.sessionIssuer.userName" as userName nodrop
| json field=_raw "userIdentity.userName" as userName nodrop
| json field=_raw "errorCode" as error_code nodrop
| json field=_raw "errorMessage" as error_msg nodrop
| where eventSource="kinesis.amazonaws.com"
| count by error_code, error_msg, eventName, userName, sourceIPAddress