Skip to main content
Sumo Logic

Collect Logs and Metrics for the Amazon EKS - Control Plane App

This page has instructions for collecting logs and metrics for the Sumo App for Amazon EKS - Control Plane.

Collection process  

Configuring logs and metrics for the Amazon EKS - Control Plane App is a two step process:

  • Setting up collection and installing the Sumo Logic Kubernetes App.
  • Configuring CloudWatch log collection.

Step 1. Set up and install the Kubernetes App 

The Sumo Logic Kubernetes App provides the services for managing and monitoring Kubernetes worker nodes. You must set up collection and install the Kubernetes App before configuring collection for the EKS - Control Plane App. You will configure log and metric collection during this process.

To set up and install the Kubernetes app, follow the instructions in this document.

Step 2. Configure CloudWatch log collection 

To configure Amazon CloudWatch log collection, do the following:  

  1. Follow the instructions for Collecting Logs using a CloudFormation Template
  2. Refer Amazon EKS Logs for Amazon specific details.

Sample log messages 

API Server Audit 
{
   "timestamp":1561532751495,
   "message":{
   "kind":"Event",
   "apiVersion":"audit.k8s.io/v1beta1",
   "metadata":{
      "creationTimestamp":"2019-06-26T07:05:51Z"
   },
   "level":"Metadata",
   "timestamp":"2019-06-26T07:05:51Z",
   "auditID":"8c7f04e6-19ae-4b02-a3a1-c1e03bea7f98",
   "stage":"ResponseComplete",
   "requestURI":"/api/v1/namespaces/kube-system/secrets/kube-proxy-token-w7wkr",
   "verb":"get",
   "user":{
      "username":"system:apiserver",
      "uid":"bf2d8ee6-319d-4735-94a1-2903bcef27cf",
      "groups":[
         "system:masters"
      ]
   },
   "sourceIPs":[
      "127.0.0.1"
   ],
   "objectRef":{
      "resource":"secrets",
      "namespace":"kube-system",
      "name":"kube-proxy-token-w7wkr",
      "apiVersion":"v1"
   },
   "responseStatus":{
      "metadata":{
      },
      "code":200
   },
   "requestReceivedTimestamp":"2019-06-26T07:05:51.447627Z",
   "stageTimestamp":"2019-06-26T07:05:51.450399Z",
   "annotations":{
      "authorization.k8s.io/decision":"allow",
      "authorization.k8s.io/reason":""
   }
   }
} 
 Authenticator
{
   "timestamp":1561533513014,
   "message":"time=\"2019-06-26T07:18:27Z\" level=info msg=\"access granted\"
   arn=\"arn:aws:iam::956882708938:role/arun-k8s-worker-nodes-NodeInstanceRole-1Q2W9LCWIMWT3\"
   client=\"127.0.0.1:58464\" groups=\"[system:bootstrappers system:nodes]\" method=POST
   path=/authenticate uid=\"heptio-authenticator-aws:956882708938:AROA55SVHNHFL55HJ3F5S\"
   username=\"system:node:ip-192-168-222-214.ec2.internal\""
}
API Server
{
   "timestamp":1561543835000,
   "message":"I0626 10:10:35.292107   1 get.go:245] Starting watch for /api/v1/persistentvolumes,
   rv=4220807 labels= fields= timeout=5m2s"
}
Controller Manager
{"timestamp":1561544407000,"message":"I0626 10:20:07.755497       1
cronjob_controller.go:173] Unable to update status for default/sumologic-k8s-api (rv = 6489402):
Operation cannot be fulfilled on cronjobs.batch \"sumologic-k8s-api\": the object has been modified;
please apply your changes to the latest version and try again"}
Scheduler
{"timestamp":1561106587000,"message":"I0621 08:43:07.395400       1
scheduler.go:197] Failed to schedule pod: hello-app/frontend-56f7975f44-8sgj7"}

Query sample

The following query sample is taken from the Top 10 URLs with Problem Status Codes panel on the EKS - API Server Audit Overview dashboard.

_sourceCategory = "EKS_LOGS"
and _sourceName = kube-apiserver-audit*
| json field=_raw "message.responseStatus.code" as status_code
| json field=_raw "message.verb" as method
| json field=_raw "message.requestURI" as url
| json field=_raw "message.objectRef.resource" as k8_resource
| json field=_raw "message.sourceIPs" as ip
| where !(status_code matches "2*")
| count as urls_by_status by status_code, url
| sort by urls_by_status
| limit 10