Skip to main content
Sumo Logic

Create a AWS Cost Explorer Collector Source

To create an AWS Cost Explorer collector source, you have to add it to an existing Sumo Logic hosted collector. And before creating the source, identify the hosted collector you want to use or simply create a new hosted collector.

When you create an AWS Cost Explorer collector Source, you add it to an existing Sumo Logic hosted collector. Before creating the Source, you'll need to add a Field-in-Field SchemaField Extraction Rules, and identify or create the hosted collector you want to use. For further instructions, see Create a Hosted Collector.

Field-in-Field Schema

Log in to Sumo Logic, go to Manage Data > Logs > Fields. Search for the account and linkedaccount field. If not present, create it. Learn how to create and manage fields here.

Field Extraction Rules

Create a Field Extraction Rule (FER) for AWS Cost Explorer Logs. Learn how to create a Field Extraction Rule here.

  • Rule Name: AWSCostExplorerFER
  • Applied at: Ingest Time
  • Scope (Specific Data): 
account = * region CostUsd CostType StartDate EndDate MetricType Granularity Service LinkedAccount
  • Parse Expression: Enter a parse expression to create an “account” field that maps to the alias you set for each sub account. For example, if you used the “securityprod” alias for an AWS account with ID "123456789" and the “infraprod” alias for an AWS account with ID "987654321", your parse expression would look like:
json "LinkedAccount"
| if (LinkedAccount = "123456789",  "securityprod", LinkedAccount ) as LinkedAccount
| if (LinkedAccount = "987654321",  "infraprod", LinkedAccount ) as LinkedAccount

Configure an AWS Cost Explorer Source

  1. On the Manage Data > Collection > Collection page, click Add Source next to a Hosted Collector.
  2. Select AWS Cost Explorer.
    clipboard_e3121be89745cf4b0244b0df9330ebc8c.png
  3. Enter a Name for the Source in the Sumo Logic console. The Description is optional.
    cost-explorer-v2-1-1.png

  4. For Source Category (Optional), enter any string to tag the output collected from the Source. Category metadata is stored in a searchable field called _sourceCategory.

  5. For Fields, click the +Add link to add custom log metadata. Define the fields you want to associate, each field needs a name (key) and value. 

    • green check circle.png A green circle with a checkmark is shown when the field exists and is enabled in the Fields table schema.

    • orange exclamation point.png An orange triangle with an exclamation point is shown when the field doesn't exist, or is disabled, in the Fields table schema. In this case, an option to automatically add or enable the nonexistent fields to the Fields table schema is provided. If a field is sent to Sumo Logic that does not exist in the Fields schema or is disabled it is ignored, known as dropped.

It is preferable to add an account field (for the dashboards) and assign it a friendly name to identify the corresponding AWS account.
clipboard_e53375638a2d1bab3583f63821c0f6da3.png

  1. For the AWS Access Key and AWS Secret Key, provide the IAM User access key and secret key you want to use to authenticate collection requests.

  2. For the Enable Regions field, provide the regions which need to be monitored for cost. The cost incurred across these regions will be fetched separately. The region list here includes all the standard AWS regions along with “global”. “Global” region includes services like Amazon CloudFront, Amazon Route 53, Amazon IAM. If the field is left empty (default behavior), then data will be fetched from all the enabled regions of the respective AWS account. It is recommended to provide only the regions which are actively used and need to be monitored for cost. This will save the AWS cost for running this source on unused regions.

  3. For the Cost Type, provide supported cost types / MetricTypes. For details on the CostType, see Amazon's Understanding your AWS Cost Datasets: A Cheat Sheet.

    • AmortizedCost

    • BlendedCost

    • NetAmortizedCost

    • NetUnblendedCost

    • UnblendedCost

  4. For Granularity, provide 2 supported granularities for each of the MetricTypes (or cost types):

    • Daily Costs (Polled every 12h)

    • Monthly Costs (Polled every day)

  5. Add Processing Rules

  6. Click Submit when complete.

States

The AWS Cost Explorer Source reports errors, its health, and initialization status. Other than indicating that the source is healthy, you are also informed, in real-time, if the source is running into trouble communicating with AWS API, or if there's an error that requires user action indicated by Sumo Logic Health Events.

An AWS Cost Explorer Source goes through the following states when created:

  1. Pending: Once the Source is submitted, details are stored and the source is placed in a Pending state.
  2. Started: A collection task is created on the hosted collector.
  3. Initialized: Task configuration is complete in Sumo Logic.
  4. Authenticated: The Source has successfully authenticated with AWS
  5. Collecting: The Source is actively collecting data from AWS accounts.

If the Source has any issues during any one of these states, it is placed in an Error state.

Health and Status columns.png

Hover your mouse over the status icon to view a tooltip with details on the detected issue.

error status.png

When you delete the source it is placed in a Stopping state, when it has successfully stopped it is deleted from your Hosted Collector.

On the Collection page, the Health and Status for Sources is displayed. Use Health Events to investigate issues with collection.

Error types

When Sumo Logic detects an issue it is tracked by Health Events. The following table shows the three possible error types, the reason the error would occur, if the Source attempts to retry, and the name of the event log in the Health Event Index.

Type

Reason

Retries

Retry Behavior

Health Event Name

ThirdPartyConfig

Normally due to an invalid configuration. You'll need to review your Source configuration and make an update.

No retries are attempted until the Source is updated.

Not applicable

ThirdPartyConfigError

ThirdPartyGeneric

Normally due to an error communicating with the third party service APIs.

Yes

The Source will retry for up to 90 minutes, after which retries will be attempted every 60 minutes.

ThirdPartyGenericError

FirstPartyGeneric

Normally due to an error communicating with the internal Sumo Logic APIs.

Yes

The Source will retry for up to 90 minutes, after which retries will be attempted every 60 minutes.

FirstPartyGenericError

JSON Configuration

Sources can be configured using UTF-8 encoded JSON files with the Collector Management API. See how to use JSON to configure Sources for details.

Parameter

Type

Required

Description

Access

config

JSON Object

Yes

Contains the configuration parameters for the Source.

 

schemaRef

JSON Object

Yes

Use {"type":"AWS Cost Explorer"}  for an AWS Cost Explorer Source.

Not modifiable

sourceType

String

Yes

Use Universal for an AWS Cost Explorer  Source.

 

Not modifiable

The following table shows the config parameters for an AWS Cost Explorer Source.

Parameter

Type

Required

Default

Description

Access

name 

String

Yes

 

Type the desired name of the Source. The name must be unique per Collector. This value is assigned to the metadata field _source.

 

modifiable

description

String

No

Null

Type a description of the Source.

modifiable

category

String

No

Null

Type a category of the source. This value is assigned to the metadata field _sourceCategory. See best practices for details.

modifiable

fields

JSON Object

No

 

JSON map of key-value fields (metadata) to apply to the Collector or Source.

Use the string field account to tag the logs with friendly aws account name.

 

modifiable

accessID

String

Yes

 

Provide the AWS IAM User access key ID you want to use to authenticate collection requests.

modifiable

accessKey

String

Yes

 

Provide the AWS Secret Key you want to use to authenticate collection requests.

modifiable

granularity

String array

Yes

 

Provide a list, such as ["daily","monthly"]

modifiable

costMetrics

String array

Yes

 

Provide a list, such as
["AmortizedCost","BlendedCost","NetAmortizedCost",
"NetUnblendedCost","UnblendedCost"]

modifiable

AWS Region

String array

No

 

Provide a list, such as
["US East (Ohio)","US West (Oregon)"] 

modifiable

AWS Cost Explorer Source JSON Example:

{
  "api.version":"v1",
  "source":{
    "schemaRef":{
      "type":"AWS Cost Explorer"
    },
    "config":{
      "accessID":"********",
      "name":"billing200",
      "description":"billing200",
      "fields":{
        "_siemForward":false,
        "account":"prod"
      },
      "accessKey":"********",
      "granularity":["daily","monthly"],
      "costMetrics":["AmortizedCost","BlendedCost","NetAmortizedCost","NetUnblendedCost","UnblendedCost"],
      "category":"aws/billing"
    },
    "state":{
      "state":"Collecting"
    },
    "sourceType":"Universal"
  }
}