Amazon ECS without Container Insights and Traces

Amazon Elastic Container Service (Amazon ECS) is a container management service that allows you to manage Docker containers on a cluster of Amazon EC2 instances. The Sumo Logic app for Amazon ECS provides preconfigured searches and Dashboards that allow you to monitor various metrics (CPU and Memory Utilization, CPU and Memory Reservation) across ECS clusters and services. The app also monitors API calls made by or on behalf of Amazon ECS in your AWS account.

We offer two different ECS versions, which have separate data collection steps:

Collect Logs and Metrics for ECS. This version collects ECS CloudWatch Metrics and ECS Events using AWS CloudTrail.
Collect Logs, Metrics (Container Insights+CloudWatch) and Traces for ECS. This version collects ECS CloudWatch Metrics, Container Insights Metrics, ECS Events using AWS CloudTrail, Application Logs and Traces. Metrics collected by Container Insights are charged as custom metrics. For more information about CloudWatch pricing, see Amazon CloudWatch Pricing. This solution enables you to monitor both EC2 and Fargate based ECS deployments. For instructions on collecting this data, refer to the Amazon Elastic Container Service (ECS) using Container Insights and CloudWatch.

This page has instructions for collecting logs and metrics for the Amazon ECS without Container Insights and Traces app. It uses the following data:

CloudWatch Metrics
AWS CloudTrail Events

Sample log messages

Click to expand

{
   "eventVersion":"1.04",
   "userIdentity":{
      "type":"AssumedRole",
      "principalId":"ADFDDDFF7FDF7GFFF2DF0:i-76vfa923",
      "arn":"arn:aws:sts::435456556566:assumed-role/ecsInstanceRole/i-76vfa923",
      "accountId":"435456556566",
      "accessKeyId":"AOFGPJFIJFFOIJFIOJHF",
      "sessionContext":{
         "attributes":{
            "mfaAuthenticated":"false",
            "creationDate":"2017-10-02T20:08:54.107Z"
         },
         "sessionIssuer":{
            "type":"Role",
            "principalId":"ADFDDDFF7FDF7GFFF2DF0",
            "arn":"arn:aws:iam::435456556566:role/ecsInstanceRole",
            "accountId":"435456556566",
            "userName":"kevin"
         }
      }
   },
   "eventTime":"2017-10-02T20:08:54.107Z",
   "eventSource":"ecs.amazonaws.com",
   "eventName":"RegisterTaskDefinition",
   "awsRegion":"us-west-1",
   "sourceIPAddress":"73.168.34.72",
   "userAgent":"Amazon ECS Agent - v1.12.2 (ecda8a6) (+http://aws.amazon.com/ecs/)",
   "requestParameters":{
      "attributes":[
         {
            "name":"com.amazonaws.ecs.capability.privileged-container"
         },
         {
            "name":"com.amazonaws.ecs.capability.docker-remote-api.1.17"
         },
         {
            "name":"com.amazonaws.ecs.capability.docker-remote-api.1.18"
         },
         {
            "name":"com.amazonaws.ecs.capability.docker-remote-api.1.19"
         },
         {
            "name":"com.amazonaws.ecs.capability.docker-remote-api.1.20"
         },
         {
            "name":"com.amazonaws.ecs.capability.docker-remote-api.1.21"
         },
         {
            "name":"com.amazonaws.ecs.capability.docker-remote-api.1.22"
         },
         {
            "name":"com.amazonaws.ecs.capability.logging-driver.json-file"
         },
         {
            "name":"com.amazonaws.ecs.capability.logging-driver.syslog"
         },
         {
            "name":"com.amazonaws.ecs.capability.logging-driver.awslogs"
         },
         {
            "name":"com.amazonaws.ecs.capability.ecr-auth"
         },
         {
            "name":"com.amazonaws.ecs.capability.task-iam-role"
         },
         {
            "name":"com.amazonaws.ecs.capability.task-iam-role-network-host"
         }
      ],
      "totalResources":[
         {
            "type":"INTEGER",
            "doubleValue":0.0,
            "integerValue":1024,
            "longValue":0,
            "name":"CPU"
         },
         {
            "type":"INTEGER",
            "doubleValue":0.0,
            "integerValue":995,
            "longValue":0,
            "name":"MEMORY"
         },
         {
            "type":"STRINGSET",
            "stringSetValue":[
               "22",
               "2375",
               "2376",
               "51678",
               "51679"
            ],
            "doubleValue":0.0,
            "integerValue":0,
            "longValue":0,
            "name":"PORTS"
         },
         {
            "type":"STRINGSET",
            "stringSetValue":[ ],
            "doubleValue":0.0,
            "integerValue":0,
            "longValue":0,
            "name":"PORTS_UDP"
         }
      ],
      "instanceIdentityDocumentSignature":"pqWe1trtreertermhC6vz\nZ0e/ZyOVVKXOb0fiiouyuyturtyreuFaoghqQ0wWurXzcHb6CrtreyteV6hPM=",
      "cluster":"graphite",
      "instanceIdentityDocument":"{\n  \"privateIp\" : \"10.0.1.83\",\n  \"devpayProductCodes\" : null,\n  \"availabilityZone\" : \"us-west-1c\",\n  \"accountId\" : \"435456556566\",\n  \"version\" : \"2010-08-31\",\n  \"instanceId\" : \"i-76vfa923\",\n  \"billingProducts\" : null,\n  \"instanceType\" : \"t2.micro\",\n  \"imageId\" : \"ami-444d0224\",\n  \"pendingTime\" : \"2016-11-15T21:07:08Z\",\n  \"architecture\" : \"x86_64\",\n  \"kernelId\" : null,\n  \"ramdiskId\" : null,\n  \"region\" : \"us-west-1\"\n}"
   },
   "responseElements":{
      "containerInstance":{
         "versionInfo":{ },
         "runningTasksCount":0,
         "ec2InstanceId":"i-13dcar4566",
         "remainingResources":[
            {
               "type":"INTEGER",
               "doubleValue":0.0,
               "integerValue":1024,
               "longValue":0,
               "name":"CPU"
            },
            {
               "type":"INTEGER",
               "doubleValue":0.0,
               "integerValue":995,
               "longValue":0,
               "name":"MEMORY"
            },
            {
               "type":"STRINGSET",
               "stringSetValue":[
                  "22",
                  "2376",
                  "2375",
                  "51678",
                  "51679"
               ],
               "doubleValue":0.0,
               "integerValue":0,
               "longValue":0,
               "name":"PORTS"
            },
            {
               "type":"STRINGSET",
               "stringSetValue":[ ],
               "doubleValue":0.0,
               "integerValue":0,
               "longValue":0,
               "name":"PORTS_UDP"
            }
         ],
         "agentConnected":true,
         "pendingTasksCount":0,
         "registeredResources":[
            {
               "type":"INTEGER",
               "doubleValue":0.0,
               "integerValue":1024,
               "longValue":0,
               "name":"CPU"
            },
            {
               "type":"INTEGER",
               "doubleValue":0.0,
               "integerValue":995,
               "longValue":0,
               "name":"MEMORY"
            },
            {
               "type":"STRINGSET",
               "stringSetValue":[
                  "22",
                  "2376",
                  "2375",
                  "51678",
                  "51679"
               ],
               "doubleValue":0.0,
               "integerValue":0,
               "longValue":0,
               "name":"PORTS"
            },
            {
               "type":"STRINGSET",
               "stringSetValue":[ ],
               "doubleValue":0.0,
               "integerValue":0,
               "longValue":0,
               "name":"PORTS_UDP"
            }
         ],
         "containerInstanceArn":"arn:aws:ecs:us-west-1:435456556566:container-instance/3f28c319-u9n2-1476-3d2n-b7c254fv411",
         "attributes":[
            {
               "name":"com.amazonaws.ecs.capability.privileged-container"
            },
            {
               "name":"com.amazonaws.ecs.capability.docker-remote-api.1.17"
            },
            {
               "name":"com.amazonaws.ecs.capability.docker-remote-api.1.18"
            },
            {
               "name":"com.amazonaws.ecs.capability.docker-remote-api.1.19"
            },
            {
               "name":"com.amazonaws.ecs.capability.docker-remote-api.1.20"
            },
            {
               "name":"com.amazonaws.ecs.capability.docker-remote-api.1.21"
            },
            {
               "name":"com.amazonaws.ecs.capability.docker-remote-api.1.22"
            },
            {
               "name":"com.amazonaws.ecs.capability.logging-driver.json-file"
            },
            {
               "name":"com.amazonaws.ecs.capability.logging-driver.syslog"
            },
            {
               "name":"com.amazonaws.ecs.capability.logging-driver.awslogs"
            },
            {
               "name":"com.amazonaws.ecs.capability.ecr-auth"
            },
            {
               "name":"com.amazonaws.ecs.capability.task-iam-role"
            },
            {
               "name":"com.amazonaws.ecs.capability.task-iam-role-network-host"
            }
         ],
         "status":"ACTIVE",
         "version":1
      }
   },
   "requestID":"ae86b372-ab77-11e6-824c-c7c4220f0423",
   "eventID":"ff9fc985-1fbe-4717-965b-607dda32f620",
   "eventType":"AwsApiCall",
   "recipientAccountId":"435456556566"
}

Sample queries

Deleted Resources Over Time
_sourceCategory=ecs* (DeleteCluster or DeleteService or DeregisterContainerInstance or DeregisterTaskDefinition or StopTask) and !(InternalFailure)
| json "eventName" as event_name
| parse "\"userName\":\"*\"" as user
| parse "\"awsRegion\":\"*\"" as region
| parse "\"cluster\":\"*\"" as cluster
| timeslice 1h
| parse regex field=event_name "^(?:Delete|Deregister|Stop)(?<resource_type>[A-Z][A-Za-z]+)"
| count by resource_type, _timeslice
| transpose row _timeslice column resource_type

Creating Fields in Field Schema

Classic UI. In the main Sumo Logic menu, select Manage Data > Logs > Fields.
New UI. In the top menu select Configuration, and then under Logs select Fields. You can also click the Go To... menu at the top of the screen and select Fields.
Search for the following fields: account, namespace, region field.
If not present, create it. Learn how to create and manage fields here.

Creating Field Extraction Rule(s)

Create a Field Extraction Rule for CloudTrail Logs (learn more).

Rule Name: AwsObservabilityECSCloudTrailLogsFER
Applied at: Ingest Time
Scope (Specific Data):
account=* eventname eventsource "ecs.amazonaws.com"
Parse Expression:
| json "eventSource", "awsRegion", "requestParameters.tableName", "recipientAccountId" as eventSource, region, tablename, accountid nodrop
| where eventSource = "ecs.amazonaws.com"
| "aws/ecs" as namespace
| fields region, namespace, accountid

Collect Logs and Metrics for Amazon ECS

This section has instructions for collecting logs and metrics for the Amazon ECS app.

Collect Metrics for Amazon ECS

Sumo Logic supports collecting metrics using two source types:
- Configure an AWS Kinesis Firehose for Metrics Source (recommended) or
- Configure an Amazon CloudWatch Source for Metrics
note
Amazon ECS metrics use the AWS/ECS namespace
Metadata. Click the +Add Field link to add custom log metadata fields. Define the fields you want to associate, each field needs a name (key) and value.
1. Add an account field and assign it a value which is a friendly name / alias to your AWS account from which you are collecting logs. Logs can be queried via the “account field”.
2. Keep in mind:
  - A green circle with a check mark is shown when the field exists and is enabled in the Fields table schema.
  - An orange triangle with an exclamation point is shown when the field doesn't exist, or is disabled, in the Fields table schema. In this case, an option to automatically add or enable the nonexistent fields to the Fields table schema is provided. If a field is sent to Sumo Logic that does not exist in the Fields schema or is disabled it is ignored, known as dropped.

Collect ECS events using CloudTrail

Configure a AWS CloudTrail Logs Source.
Metadata. Click the +Add Field link to add custom log metadata Fields. Define the fields you want to associate, each field needs a name (key) and value.
1. Add an account field and assign it a value which is a friendly name / alias to your AWS account from which you are collecting logs. Logs can be queried via the “account field”.
2. Keep in mind:
  - A green circle with a check mark is shown when the field exists and is enabled in the Fields table schema.
  - An orange triangle with an exclamation point is shown when the field doesn't exist, or is disabled, in the Fields table schema. In this case, an option to automatically add or enable the nonexistent fields to the Fields table schema is provided. If a field is sent to Sumo Logic that does not exist in the Fields schema or is disabled it is ignored, known as dropped.

Installing the Amazon ECS app

Now that you have set up collection for Amazon ECS, install the Sumo Logic app for Amazon ECS to use the pre-configured searches and dashboards that provide visibility into your environment for real-time analysis of overall usage.

To install the app:

Select App Catalog.
In the 🔎 Search Apps field, run a search for your desired app, then select it.
Click Install App.
note
Sometimes this button says Add Integration.
On the next configuration page, under Select Data Source for your App, complete the following fields:
- Data Source. Select one of the following options:
  - Choose Source Category and select a source category from the list; or
  - Choose Enter a Custom Data Filter, and enter a custom source category beginning with an underscore. For example, _sourceCategory=MyCategory.
- Folder Name. You can retain the existing name or enter a custom name of your choice for the app.
- All Folders (optional). The default location is the Personal folder in your Library. If desired, you can choose a different location and/or click New Folder to add it to a new folder.
Click Next.
Look for the dialog confirming that your app was installed successfully.

Post-installation

Once your app is installed, it will appear in your Personal folder or the folder that you specified. From here, you can share it with other users in your organization. Dashboard panels will automatically start to fill with data matching the time range query received since you created the panel. Results won't be available immediately, but within about 20 minutes, you'll see completed graphs and maps.

Viewing the Amazon ECS app dashboards

All dashboards have a set of filters that you can apply to the entire dashboard. Use these filters to drill down and examine the data to a granular level.

You can change the time range for a dashboard or panel by selecting a predefined interval from a drop-down list, choosing a recently used time range, or specifying custom dates and times. Learn more.
You can use template variables to drill down and examine the data on a granular level. For more information, see Filtering Dashboards with Template Variables.
Most Next-Gen apps allow you to provide the scope at the installation time and are comprised of a key (_sourceCategory by default) and a default value for this key. Based on your input, the app dashboards will be parameterized with a dashboard variable, allowing you to change the dataset queried by all panels. This eliminates the need to create multiple copies of the same dashboard with different queries.

Overview

The Amazon ECS - Overview dashboard provides an overview of CPU and memory utilization across all your ECS clusters and services. You can determine which services are high in utilization and accordingly make decisions for the ECS deployment.

Use this dashboard to:

Quickly determine resource utilization across all your ECS clusters and services so as to provision more capacity or optimize on cost.
Get a quick snapshot of overall ECS cluster health, including CPU and memory utilization and reservation levels.
Identify clusters or services with unusually high or low resource usage or reservation percentages.
View details of individual clusters and services, including their regions and associated accounts.

Audit Events

The Amazon ECS - Audit Events dashboard provides insights into changes to your ECS environment including top IAM users, locations of events. The dashboard also shows the created, updated, and deleted events with respect to time, along with the details for the top 10 AWS Identity and Access Management users, and the last 20 Container Registration and Deregistration Events.

Use this dashboard to:

Quickly identify all changes to your ECS environment.
Monitor locations from which changes are being made locations.
Examine details and trends for created, updated and deleted ECS resources.
Investigate specific container registration and deregistration events in different regions and clusters.

Resource Utilization

The Amazon ECS - Resource Utilization dashboard provides trends around CPU and Memory utilization for clusters and services.

Cluster CPU or Cluster memory utilization metrics are only used for tasks using the EC2 launch type.
Service CPU or service memory utilization metrics are used for tasks using both the Fargate and the EC2 launch type.

Use this dashboard to:

Monitor real-time CPU and memory usage across your ECS clusters and services.
Identify performance bottlenecks or underutilized resources in your ECS environment.
Compare utilization patterns between clusters and individual services to optimize resource allocation.

Resource Reservation

The Amazon ECS - Resource Reservation dashboard provides detailed insights into the average reservation (units utilized) by CPU, Memory, and GPU for a given cluster.

These metrics are available for clusters only.
This metric is used only on clusters with tasks or services using the EC2 launch type. It's not supported on clusters with tasks using the Fargate launch type.

Use this dashboard to:

Track average resource reservation levels across different ECS clusters and services.
Identify potential resource constraints or overprovisioning in your ECS environment.
Compare reservation patterns between different types of resources (CPU, memory, GPU) over time.

Sample log messages​

Sample queries​

Creating Fields in Field Schema​

Creating Field Extraction Rule(s)​

Collect Logs and Metrics for Amazon ECS​

Collect Metrics for Amazon ECS​

Collect ECS events using CloudTrail​

Installing the Amazon ECS app​

Viewing the Amazon ECS app dashboards​

Overview​

Audit Events​

Resource Utilization​

Resource Reservation​

Sample log messages

Sample queries

Creating Fields in Field Schema

Creating Field Extraction Rule(s)

Collect Logs and Metrics for Amazon ECS

Collect Metrics for Amazon ECS

Collect ECS events using CloudTrail

Installing the Amazon ECS app

Viewing the Amazon ECS app dashboards

Overview

Audit Events

Resource Utilization

Resource Reservation