Skip to main content
Sumo Logic

Collect Logs for Amazon SES

Steps to collect logs from Amazon SES, and to ingest them into Sumo.

This page has instructions for collecting CloudTrail events and SES notifications for the Sumo App for Amazon SES.

Step 1: Configure CloudTrail

  1. Enable CloudTrail in your AWS account. You'll be offered the option to create an S3 bucket.
  2. Grant Sumo Logic access to the Amazon S3 bucket created or selected above.

Step 2: Collect Amazon SES events using CloudTrail

These configuration instructions apply to log collection from all AWS Source types. Select the correct Source type for your Source in Step 3.

  1. In Sumo Logic select Manage Data > Collection > Collection
  2. On the Collectors page, click Add Source next to a Hosted Collector, either an existing Hosted Collector, or one you have created for this purpose.
  3. Select your AWS Source type.
  4. Enter a name for the new Source. A description is optional.
  5. For Bucket Name, enter the exact name of your organization's S3 bucket. Be sure to double-check the name as it appears in AWS, for example:
    S3_Bucket_name.png
  6. For Path Expression, enter the wildcard pattern that matches the S3 objects you'd like to collect. You can use one wildcard (*) in this string. Recursive path expressions use a single wildcard and do NOT use a leading forward slash. See About Amazon Path Expressions for details.
  7. Collection should begin. Choose or enter how far back you'd like to begin collecting historical logs. You can either:
    • Choose a predefined value from dropdown list, ranging from "Now" to “72 hours ago” to “All Time”, or
    • Enter a relative value. To enter a relative value, click the Collection should begin field and press the delete key on your keyboard to clear the field. Then, enter a relative time expression, for example -1w. You can define when you want collection to begin in terms of months (M), weeks (w), days (d), hours (h) and minutes (m).
  8. For Source Category, enter any string to tag the output collected from this Source. (Category metadata is stored in a searchable field called _sourceCategory.)
  9. For AWS Access you have two Access Method options. Select Role-based access or Key access based on the AWS authentication you are providing. Role-based access is preferred, this was completed in the prerequisite step Grant Sumo Logic access to an AWS Product.
    • For Role-based access enter the Role ARN that was provided by AWS after creating the role. 
      Role based access input roleARN.png
    • For Key access enter the Access Key ID and Secret Access Key. See AWS Access Key ID and AWS Secret Access Key for details.
  10. Log File Discovery. You have the option to set up Amazon Simple Notification Service (SNS) to notify Sumo Logic of new items in your S3 bucket. A scan interval is required and automatically applied to detect log files.
    • SNS Subscription Endpoint (Optional). New files will be collected by Sumo Logic as soon as the notification is received. This will provide faster collection versus having to wait for the next scan to detect the new file.
      1. To set up the subscription you need to get an endpoint URL from Sumo to provide to AWS. This process will save your Source and begin scanning your S3 bucket when the endpoint URL is generated. Click on Create URL and use the provided endpoint URL when creating your subscription in step C.
        log file discovery input.png

Set up SNS in AWS (Optional)
  1. Go to Services > Simple Notification Service and click Create Topic. Enter a Topic name and click Create topic. Copy the provided Topic ARN, you’ll need this for the next step.

  2. Again go to Services > Simple Notification Service and click Create Subscription. Paste the Topic ARN from step B above. Select HTTPS as the protocol and enter the Endpoint URL provided while creating the S3 source in Sumo Logic. Click Create subscription and a confirmation request will be sent to Sumo Logic. The request will be automatically confirmed by Sumo Logic.

  3. Select the Topic created in step B and navigate to Actions > Edit Topic Policy. Use the following policy template, replace the SNS-topic-ARN and bucket-name placeholders in the Resource section of the JSON policy with your actual SNS topic ARN and S3 bucket name:

    {
        "Version": "2008-10-17",
        "Statement": [{
            "Effect": "Allow",
            "Principal": {
                "AWS": "*"
            },
            "Action": [
                "SNS:Publish"
            ],
            "Resource": "SNS-topic-ARN",
            "Condition": {
                "ArnLike": {
                    "aws:SourceArn": "arn:aws:s3:*:*:bucket-name"
                }
            }
        }]
    }

  4. Go to Services > S3 and select the bucket to which you want to attach the notifications. Navigate to Properties > Events > Add Notification. Enter a Name for the event notification. In the Events section select ObjectCreate (All). In the Send to section (notification destination) select SNS Topic. An SNS section becomes available, select the name of the topic you created in step B from the dropdown. Click Save.

Complete set up in Sumo
  • Scan Interval. Sumo Logic will periodically scan your S3 bucket for new items in addition to SNS notifications. Automatic is recommended to not incur additional AWS charges. This sets the scan interval based on if subscribed to an SNS topic endpoint and how often new files are detected over time. You may enter a set frequency to scan your S3 bucket for new data. To learn more about Scan Interval considerations, see About setting the S3 Scan Interval.
  1. Set any of the following under Advanced:
  • Enable Timestamp Parsing. This option is selected by default. If it's deselected, no timestamp information is parsed at all.
    • Time Zone. There are two options for Time Zone. You can use the time zone present in your log files, and then choose an option in case time zone information is missing from a log message. Or, you can have Sumo Logic completely disregard any time zone information present in logs by forcing a time zone. It's very important to have the proper time zone set, no matter which option you choose. If the time zone of logs can't be determined, Sumo Logic assigns logs UTC; if the rest of your logs are from another time zone your search results will be affected.
    • Timestamp Format. By default, Sumo Logic will automatically detect the timestamp format of your logs. However, you can manually specify a timestamp format for a Source. See Timestamps, Time Zones, Time Ranges, and Date Formats for more information.
  • Enable Multiline Processing. See Collecting Multiline Logs for details on multiline processing and its options. This is enabled by default. Use this option if you're working with multiline messages (for example, log4J or exception stack traces). Deselect this option if you want to avoid unnecessary processing when collecting single-message-per-line files (for example, Linux system.log). Choose one of the following:  
    • Infer Boundaries. Enable when you want Sumo Logic to automatically attempt to determine which lines belong to the same message. If you deselect the Infer Boundaries option, you will need to enter a regular expression in the Boundary Regex field to use for detecting the entire first line of multiline messages.
    • Boundary Regex. You can specify the boundary between messages using a regular expression. Enter a regular expression that matches the entire first line of every multiline message in your log files.
  1. Create any Processing Rules you'd like for the AWS Source.
  2. When you are finished configuring the Source click Save.

Step 3: Configure an HTTP source for SES notifications

In this step, you set up an  HTTP source to receive SES notifications.

  • Name: Required
  • Source Category: AWS/SES/Notifications.
  • Timestamp Parsing Settings:
    • Enable Timestamp Parsing: True
    • Timezone:  Logs are sent in UTC by default and can be auto detected
    • Timestamp Format: Auto Detect
  • Enable One Message Per Request: True

Sample Log Message

CloudTrail log

{ "eventVersion":"1.04", "userIdentity":{ "type":"IAMUser", "principalId":"AIDAI1234567890YGJ2G6", "arn":"arn:aws:iam::123456789033:user/mkmiller", "accountId":"123456789033", "accessKeyId":"ASI1234567890IHSAOIQ", "userName":"jbrown", "sessionContext":{ "attributes":{ "mfaAuthenticated":"true", "creationDate":"2017-12-12T11:18:58Z" } }, "invokedBy":"signin.amazonaws.com" }, "eventTime":"2018-01-02T19:45:18Z", "eventSource":"ses.amazonaws.com", "eventName":"GetIdentityMailFromDomainAttributes", "awsRegion":"us-west-3", "sourceIPAddress":"220.18.108.139", "userAgent":"signin.amazonaws.com", "requestParameters":{ "identities":[ "pwilson@sumologic.com", "amoore1@sumologic.com" ] }, "responseElements":{ "mailFromDomainAttributes":{ "mkmiller@sumologic.com":{ "behaviorOnMXFailure":"UseDefaultValue" }, "mperez1@sumologic.com":{ "behaviorOnMXFailure":"UseDefaultValue" } } }, "requestID":"9774b3e6-df4d-11e7-8e07-7d3a17657a4d", "eventID":"d36bd7a4-03f0-4245-a6b8-cdb56cfc8e91", "eventType":"AwsApiCall", "recipientAccountId":"123456789033" }

SES log

{"notificationType":"Delivery","mail":{"timestamp":"2018-02-08T18:18:09.060Z","source":"Sumo Logic <service@sumologic.com>","sourceArn":"arn:aws:ses:us-west-3:123456789029:identity/service@sumologic.com","sourceIp":"19.171.22.2","sendingAccountId":"122226337001","messageId":"010001606dc7dea0-91abab6b-b5fc-47as-921f-813c92ac40ud-000000","destination":["bob@example.com"]},"delivery":{"timestamp":"2017-12-19T07:58:23.735Z","processingTimeMillis":865,"recipients":["jason@sumo.com"],"smtpResponse":"250 2.0.0 OK 1513670303 h58si3264405qta.418 - gsmtp","remoteMtaIp":"169.107.162.237","reportingMTA":"a9-19.smtp-out.amazonses.com"}}

{"notificationType":"Bounce","bounce":\{"bounceType":"Permanent","bounceSubType":"Suppressed","bouncedRecipients":[{"emailAddress":"larry@customer.com","action":"failed","status":"5.1.1","diagnosticCode":"Amazon SES has suppressed sending to this address because it has a recent history of bouncing as an invalid address. For more information about how to remove an address from the suppression list, see the Amazon SES Developer Guide: http://docs.aws.amazon.com/ses/lates...ssionlist.html "}],"timestamp":"2018-04-12T11:46:41.807Z","feedbackId":"010001606e10a2db-3807dda0-4311-4b62-b883-8e0cb4122954-000000","reportingMTA":"dns; amazonses.com"},"mail":\{"timestamp":"2017-12-19T09:17:52.309Z","source":"Sumo Logic <service@sumologic.com>","sourceArn":"arn:aws:ses:us-east-3:123456789029:identity/service@sumologic.com","sourceIp":"169.107.162.237","sendingAccountId":"123456789029","messageId":"010001606e109e93-29782834-7101-4a4a-abbd-2d3e971d1173-000000","destination":["naren@demo.com"]}}

{"notificationType":"Complaint","complaint":{"complainedRecipients":[{"emailAddress":"nathan@sumodemoacme.com"}],"timestamp":"2018-04-12T12:25:07.641Z","feedbackId":"01000162b539f06b-d701b0a8-bde5-48ea-85b2-a8a58e4de012-000000","userAgent":"AOL SComp","complaintFeedbackType":"abuse","arrivalDate":"2018-04-12T12:25:07.641Z"},"mail":{"timestamp":"2018-04-12T12:25:07.641Z","source":"Sumo Logic Information <service@sumologic.com>","sourceArn":"arn:aws:ses:us-west-2:123456789029:identity/service@sumologic.com","sourceIp":"147.106.118.104","sendingAccountId":"123456789029","messageId":"0100016292d33f2f-6a6d0111-cfb3-499b-a667-9edae2d901c5-000000","destination":(["jackson@longsumo.com"]}}

Query Sample

Top bounced email addresses

(_sourceCategory=aws-ses or _sourceCategory=AWS/SES/Notifications) "\"notificationType\":\"Bounce\""
| json "notificationType" nodrop
| json "bounce.bounceSubType" as bounceSubType nodrop
| json "bounce.bounceType" as bounceType nodrop
| json "bounce.bouncedRecipients" as bouncedRecipients nodrop
| parse regex field=bouncedRecipients "\"emailAddress\":\"(?[^\"]*)\"" multi
| parse field=BouncedemailAddress "*@*" as name, domain
| where notificationType="Bounce"
| count as eventCount by BouncedemailAddress
| sort by eventCount, BouncedemailAddress
| limit 10