Collect logs for the AKS - Control Plane App

AKS - Control Plane Logs

The AKS - Control Plane App uses Azure managed master node logs in your Azure Kubernetes Service (AKS). The AKS - Control Plane App collects logs for the following services: 

• kube-apiserver - The API server exposes the underlying Kubernetes APIs. This component provides the interaction for management tools, such as kubectl or the Kubernetes dashboard.

• kube-scheduler - The Scheduler determines what nodes can run the workload when you create or scale applications and then starts them.

• kube-controller-manager - The Controller Manager oversees a number of smaller controllers that perform actions, such as replicating pods and handling node operations.

The Sumo Logic Kubernetes App works in conjunction with the AKS - Control Plane App and allows you to monitor worker node logs, as well as metrics for the Azure monitor and worker nodes.

Collection process

Configuring logs and metrics for the AKS - Control Plane App is a two step process:

• Setting up collection and installing the Sumo Logic Kubernetes App.
• Configuring Azure monitor and enabling the master node logs in AKS.

Step 1. Setup and Install the Sumo Logic Kubernetes App 

The Sumo Logic Kubernetes App provides the services for managing and monitoring Kubernetes worker nodes. You must set up collection and  install the Kubernetes App before configuring collection for the AKS - Control Plane App. You will configure log and metric collection during this process.

To set up and install the Kubernetes App, follow the instructions in this document.

Step 2. Configure Azure Monitor and enable logs in AKS

This section walks you through the process of configuring a pipeline to send logs from Azure Monitor to Sumo Logic. The AKS - Control Plane App utilizes the following log types: 

• kube-apiserver
• kube-scheduler
• kube-controller-manager 

To configure Azure Monitor and enable AKS logs, do the following:

1. Follow Step 1 and Step 2 on this page: Collect Logs from Azure Monitor
2. Enable the Kubernetes master node logs in Azure Kubernetes Service to send logs to an event hub created in the previous step.

Sample log messages

{
"operationName":"Microsoft.ContainerService/managedClusters/diagnosticLogs/Read",
"category":"kube-apiserver",
"resourceId":"/SUBSCRIPTIONS/C111111-DXXX-4XXX-AXXX-900000000/RESOURCEGROUPS/AG-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/AG-AKS-CLUSTER",
"properties":▼{
"log":"I0624 20:14:59.855669       1 wrap.go:47] PUT /api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=10s: 
(9.05251ms) 200 [hyperkube/v1.12.8 (linux/amd64) kubernetes/a89f8c1/leader-election 172.31.1.1:48110]
",
"stream":"stderr",
"pod":"kube-apiserver-796bd9b775-xqk5s",
"containerID":"2d6cac1300da3226323fd1b936fe8278b87cba2b7a1bbd9c8401da6f8e786f5e"
},
"time":"2019-06-24T20:14:59.000Z"
}

{
"operationName":"Microsoft.ContainerService/managedClusters/diagnosticLogs/Read",
"category":"kube-controller-manager",
"resourceId":"/SUBSCRIPTIONS/C111111-DXXX-4XXX-AXXX-900000000/RESOURCEGROUPS/AG-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/AG-AKS-CLUSTER",
"properties":▼{
"log":"I0624 07:27:25.976386       1 event.go:221] Event(v1.ObjectReference{Kind:\"DaemonSet\", 
Namespace:\"kube-system\", Name:\"kube-proxy\", UID:\"2dfb3905-7dac-11e9-b60d-0a58ac1f01f6\", 
APIVersion:\"apps/v1\", ResourceVersion:\"4150266\", FieldPath:\"\"}): type: 'Normal' 
reason: 'SuccessfulCreate' Created pod: kube-proxy-xhmv7
",
"stream":"stderr",
"pod":"kube-controller-manager-59fd65c5bd-694kh",
"containerID":"667b540db41b66e914ca2ed496e0bef6d4a0b73fc832f5d5eba958d8a56a5e93"
},
"time":"2019-06-24T07:27:25.000Z"
}

{
"operationName":"Microsoft.ContainerService/managedClusters/diagnosticLogs/Read",
"category":"kube-scheduler",
"resourceId":"/SUBSCRIPTIONS/C111111-DXXX-4XXX-AXXX-900000000/RESOURCEGROUPS/AG-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/AG-AKS-CLUSTER",
"properties":▼{
"log":"W0622 22:50:25.377565       1 reflector.go:256] k8s.io/client-go/informers/factory.go:131: 
watch of *v1.StorageClass ended with: too old resource version: 3828720 (3970094)
",
"stream":"stderr",
"pod":"kube-scheduler-744886667c-cxnvk",
"containerID":"6093b28d82e1f850fb5a9f59ed8c31aa7179e8a3907449dcd2450a6605341a60"
},
"time":"2019-06-22T22:50:25.000Z"
}

Query samples

_sourceCategory="azure/aks" "kube-apiserver"
| json "properties.log", "category", "time", "properties.pod", "resourceId" as log, category, time, pod, resourceId
| where category ="kube-apiserver"
| parse regex field=log "(?<severity>W|I|F|E)(?<tt>[\S]+) (?<times>[\d:.]+)[\s]+(?<log_msg>.*)"
| parse regex field=resourceId "RESOURCEGROUPS\/(?<resource_grp>[\S]+)\/PROVIDERS\/MICROSOFT\.CONTAINERSERVICE\/MANAGEDCLUSTERS\/(?<cluster>[\S]+)"
| timeslice 1h
| count by _timeslice, severity
| transpose row _timeslice column severity
| fillmissing timeslice(1h)

_sourceCategory="azure/aks" ("kube-controller-manager") 
| json "properties.log", "category", "time", "properties.pod", "resourceId" as log, category, time, pod, resourceId
| where category ="kube-controller-manager"
| parse regex field=log "(?<severity>W|I|F|E)(?<tt>[\S]+) (?<times>[\d:.]+)[\s]+(?<log_msg>.*)"
| parse regex field=resourceId "RESOURCEGROUPS\/(?<resource_grp>[\S]+)\/PROVIDERS\/MICROSOFT\.CONTAINERSERVICE\/MANAGEDCLUSTERS\/(?<cluster>[\S]+)"
| timeslice 1h
| count by _timeslice, severity
| transpose row _timeslice column severity
| fillmissing timeslice(1h)

_sourceCategory="azure/aks" "kube-scheduler"
| json "properties.log", "category", "time", "properties.pod", "resourceId" as log, category, time, pod, resourceId
| where category ="kube-scheduler"
| parse regex field=log "(?<severity>W|I|F|E)(?<tt>[\S]+) (?<times>[\d:.]+)[\s]+(?<log_msg>.*)"
| parse regex field=resourceId "RESOURCEGROUPS\/(?<resource_grp>[\S]+)\/PROVIDERS\/MICROSOFT\.CONTAINERSERVICE\/MANAGEDCLUSTERS\/(?<cluster>[\S]+)"
| timeslice 1h
| count by _timeslice, severity
| transpose row _timeslice column severity
| fillmissing timeslice(1h)