Collect Logs and Metrics for the Amazon Kinesis - Streams App
Steps to collect logs and metrics from Amazon Kinesis - Streams and to ingest them into Sumo.
Collect Metrics for Amazon Kinesis - Streams
- Configure a Hosted Collector.
- Configure an Amazon CloudWatch Source for Metrics.
- Name. Enter a name to display for the new Source.
- Description. Enter an optional description.
- Regions. Select your Amazon Regions for Kinesis.
- Namespaces. Select AWS/Kinesis.
- Source Category. Enter a source category. For example, kinesis_metrics.
- Access Key ID and Secret Access Key. Enter your Amazon Access Key ID and Secret Access Key.
- Scan Interval. Use the default of 5 minutes, or enter the frequency Sumo Logic will scan your CloudWatch Sources for new data.
- Click Save.
Collect Amazon Kinesis - Streams Events using CloudTrail
- To your Hosted Collector, add an AWS CloudTrail Source.
- Name. Enter a name to display for the new Source.
- Description. Enter an optional description.
- S3 Region. Select the Amazon Region for your Kinesis S3 bucket.
- Bucket Name. Enter the exact name of your Kinesis S3 bucket.
- Path Expression. Enter the string that matches the S3 objects you'd like to collect. You can use a wildcard (*) in this string. (DO NOT use a leading forward slash. See Amazon Path Expressions.)
- Source Category. Enter a source category. For example, kinesis_event.
- Access Key ID and Secret Access Key. Enter your Amazon Access Key ID and Secret Access Key.
- Scan Interval. Use the default of 5 minutes. Alternately, enter the frequency Sumo Logic will scan your S3 bucket for new data.
- Enable Timestamp Parsing. Select the check box.
- Time Zone. Select Ignore time zone from log file and instead use, and select UTC.
- Timestamp Format. Select Automatically detect the format.
- Enable Multiline Processing. Select the check box, and select Infer Boundaries.
- Click Save.
Sample Log Message
{ "eventVersion":"1.01", "userIdentity":{ "type":"IAMUser", "principalId":"EX_PRINCIPAL_ID", "arn":"arn:aws:iam::012345678910:user/Alice", "accountId":"012345678910", "accessKeyId":"vmLwWTxSQrcvzD", "userName":"Gosia" }, "eventTime":"2017-11-01T21:23:30+0000", "errorCode":"LimitExceedException", "errorMessage":"Rate exceeded for stream CWL-Kinesis under account 656757657843", "eventSource":"kinesis.amazonaws.com", "eventName":"MergeShards", "awsRegion":"us-east-2 ", "sourceIPAddress":"187.185.157.125", "userAgent":"aws-sdk-java/unknown-version Linux/x.xx", "requestParameters":{ "streamName":"GoodStream", "adjacentShardToMerge":"shardId-000000000002", "shardToMerge":"shardId-000000000001" }, "responseElements":null, "requestID":"e9f9c8eb-c757-11e3-bf1d-6948db3cd570", "eventID":"77cf0d06-ce90-42da-9576-71986fec411f" }
Query Sample
Details of errors in events
_sourceCategory=aws/kinesis* "kinesis.amazonaws.com" errorCode
| json field=_raw "eventSource", "eventName", "awsRegion", "sourceIPAddress","userAgent" nodrop
| json field=_raw "requestParameters.streamName" as streamName nodrop
| json field=_raw "userIdentity.sessionContext.sessionIssuer.userName" as userName nodrop
| json field=_raw "userIdentity.userName" as userName nodrop
| json field=_raw "errorCode" as error_code nodrop
| json field=_raw "errorMessage" as error_msg nodrop
| where eventSource="kinesis.amazonaws.com"
| count by error_code, error_msg, eventName, userName, sourceIPAddress