Skip to main content
Sumo Logic

Collect Logs and Metrics for the Amazon EKS - Control Plane App

This page has instructions for collecting logs and metrics for the Sumo App for Amazon EKS - Control Plane.

Collection process  

Configuring logs and metrics for the Amazon EKS - Control Plane App is a two step process:

  • Setting up collection and installing the Sumo Logic Kubernetes App.
  • Configuring CloudWatch log collection.

Step 1. Set up and install the Kubernetes App 

The Sumo Logic Kubernetes App provides the services for managing and monitoring Kubernetes worker nodes. You must set up collection and install the Kubernetes App before configuring collection for the EKS - Control Plane App. You will configure log and metric collection during this process.

To set up and install the Kubernetes app, follow the instructions in this document.

Step 2. Configure CloudWatch log collection 

Amazon EKS utilizes the following log types:

  • Kubernetes API server component logs (api) – The cluster API server is the control plane component that exposes the Kubernetes API. 
  • Audit (audit) – Kubernetes audit logs provide a record of the individual users, administrators, or system components that have affected your cluster. 
  • Authenticator (authenticator) – Authenticator logs are unique to Amazon EKS. These logs represent the control plane component that Amazon EKS uses for Kubernetes Role Based Access Control (RBAC) authentication using IAM credentials.
  • Controller manager (controllerManager) – The controller manager manages the core control loops that are shipped with Kubernetes.
  • Scheduler (scheduler) – The scheduler component manages when and where to run pods in your cluster.

For more details about EKS logging, refer the Amazon documentation.

To configure Amazon CloudWatch log collection, do the following:  

  1. Follow the instructions for Collecting Logs using a CloudFormation Template
  2. Refer Amazon EKS Logs for Amazon specific details.

Sample log messages 

API Server Audit 
{
   "timestamp":1561532751495,
   "message":{
   "kind":"Event",
   "apiVersion":"audit.k8s.io/v1beta1",
   "metadata":{
      "creationTimestamp":"2019-06-26T07:05:51Z"
   },
   "level":"Metadata",
   "timestamp":"2019-06-26T07:05:51Z",
   "auditID":"8c7f04e6-19ae-4b02-a3a1-c1e03bea7f98",
   "stage":"ResponseComplete",
   "requestURI":"/api/v1/namespaces/kube-system/secrets/kube-proxy-token-w7wkr",
   "verb":"get",
   "user":{
      "username":"system:apiserver",
      "uid":"bf2d8ee6-319d-4735-94a1-2903bcef27cf",
      "groups":[
         "system:masters"
      ]
   },
   "sourceIPs":[
      "127.0.0.1"
   ],
   "objectRef":{
      "resource":"secrets",
      "namespace":"kube-system",
      "name":"kube-proxy-token-w7wkr",
      "apiVersion":"v1"
   },
   "responseStatus":{
      "metadata":{
      },
      "code":200
   },
   "requestReceivedTimestamp":"2019-06-26T07:05:51.447627Z",
   "stageTimestamp":"2019-06-26T07:05:51.450399Z",
   "annotations":{
      "authorization.k8s.io/decision":"allow",
      "authorization.k8s.io/reason":""
   }
   }
} 
 Authenticator
{
   "timestamp":1561533513014,
   "message":"time=\"2019-06-26T07:18:27Z\" level=info msg=\"access granted\"
   arn=\"arn:aws:iam::956882708938:role/arun-k8s-worker-nodes-NodeInstanceRole-1Q2W9LCWIMWT3\"
   client=\"127.0.0.1:58464\" groups=\"[system:bootstrappers system:nodes]\" method=POST
   path=/authenticate uid=\"heptio-authenticator-aws:956882708938:AROA55SVHNHFL55HJ3F5S\"
   username=\"system:node:ip-192-168-222-214.ec2.internal\""
}
API Server
{
   "timestamp":1561543835000,
   "message":"I0626 10:10:35.292107   1 get.go:245] Starting watch for /api/v1/persistentvolumes,
   rv=4220807 labels= fields= timeout=5m2s"
}
Controller Manager
{"timestamp":1561544407000,"message":"I0626 10:20:07.755497       1
cronjob_controller.go:173] Unable to update status for default/sumologic-k8s-api (rv = 6489402):
Operation cannot be fulfilled on cronjobs.batch \"sumologic-k8s-api\": the object has been modified;
please apply your changes to the latest version and try again"}
Scheduler
{"timestamp":1561106587000,"message":"I0621 08:43:07.395400       1
scheduler.go:197] Failed to schedule pod: hello-app/frontend-56f7975f44-8sgj7"}

Query sample

The following query sample is taken from the Top 10 URLs with Problem Status Codes panel on the EKS - API Server Audit Overview dashboard.

_sourceCategory = "EKS_LOGS"
and _sourceName = kube-apiserver-audit*
| json field=_raw "message.responseStatus.code" as status_code
| json field=_raw "message.verb" as method
| json field=_raw "message.requestURI" as url
| json field=_raw "message.objectRef.resource" as k8_resource
| json field=_raw "message.sourceIPs" as ip
| where !(status_code matches "2*")
| count as urls_by_status by status_code, url
| sort by urls_by_status
| limit 10