Collect logs for the AKS - Control Plane App
This page provides instructions for configuring a pipeline to collect Azure managed master node logs in your Azure Kubernetes Service (AKS) to an event hub, on to an Azure function, and finally to an HTTP source on a hosted collector in Sumo Logic.
AKS - Control Plane Logs
The AKS - Control Plane App uses Azure managed master node logs in your Azure Kubernetes Service (AKS). The AKS - Control Plane App collects logs for the following services:
-
kube-apiserver - The API server exposes the underlying Kubernetes APIs. This component provides the interaction for management tools, such as kubectl or the Kubernetes dashboard.
-
kube-scheduler - The Scheduler determines what nodes can run the workload when you create or scale applications and then starts them.
-
kube-controller-manager - The Controller Manager oversees a number of smaller controllers that perform actions, such as replicating pods and handling node operations.
The Sumo Logic Kubernetes App works in conjunction with the AKS - Control Plane App and allows you to monitor worker node logs, as well as metrics for the Azure monitor and worker nodes.
Collection process
Configuring logs and metrics for the AKS - Control Plane App is a two step process:
- Setting up collection and installing the Sumo Logic Kubernetes App.
- Configuring Azure monitor and enabling the master node logs in AKS.
Step 1. Setup and Install the Sumo Logic Kubernetes App
The Sumo Logic Kubernetes App provides the services for managing and monitoring Kubernetes worker nodes. You must set up collection and install the Kubernetes App before configuring collection for the AKS - Control Plane App. You will configure log and metric collection during this process.
To set up and install the Kubernetes App, follow the instructions in this document.
Step 2. Configure Azure Monitor and enable logs in AKS
This section walks you through the process of configuring a pipeline to send logs from Azure Monitor to Sumo Logic. The AKS - Control Plane App utilizes the following log types:
- kube-apiserver
- kube-scheduler
- kube-controller-manager
To configure Azure Monitor and enable AKS logs, do the following:
- Follow Step 1 and Step 2 on this page: Collect Logs from Azure Monitor
- Enable the Kubernetes master node logs in Azure Kubernetes Service to send logs to an event hub created in the previous step.
Sample log messages
kube-apiserver
{ "operationName":"Microsoft.ContainerService/managedClusters/diagnosticLogs/Read", "category":"kube-apiserver", "resourceId":"/SUBSCRIPTIONS/C111111-DXXX-4XXX-AXXX-900000000/RESOURCEGROUPS/AG-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/AG-AKS-CLUSTER", "properties":▼{ "log":"I0624 20:14:59.855669 1 wrap.go:47] PUT /api/v1/namespaces/kube-system/endpoints/kube-scheduler?timeout=10s: (9.05251ms) 200 [hyperkube/v1.12.8 (linux/amd64) kubernetes/a89f8c1/leader-election 172.31.1.1:48110] ", "stream":"stderr", "pod":"kube-apiserver-796bd9b775-xqk5s", "containerID":"2d6cac1300da3226323fd1b936fe8278b87cba2b7a1bbd9c8401da6f8e786f5e" }, "time":"2019-06-24T20:14:59.000Z" }
kube-controller-manager
{ "operationName":"Microsoft.ContainerService/managedClusters/diagnosticLogs/Read", "category":"kube-controller-manager", "resourceId":"/SUBSCRIPTIONS/C111111-DXXX-4XXX-AXXX-900000000/RESOURCEGROUPS/AG-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/AG-AKS-CLUSTER", "properties":▼{ "log":"I0624 07:27:25.976386 1 event.go:221] Event(v1.ObjectReference{Kind:\"DaemonSet\", Namespace:\"kube-system\", Name:\"kube-proxy\", UID:\"2dfb3905-7dac-11e9-b60d-0a58ac1f01f6\", APIVersion:\"apps/v1\", ResourceVersion:\"4150266\", FieldPath:\"\"}): type: 'Normal' reason: 'SuccessfulCreate' Created pod: kube-proxy-xhmv7 ", "stream":"stderr", "pod":"kube-controller-manager-59fd65c5bd-694kh", "containerID":"667b540db41b66e914ca2ed496e0bef6d4a0b73fc832f5d5eba958d8a56a5e93" }, "time":"2019-06-24T07:27:25.000Z" }
kube-scheduler
{ "operationName":"Microsoft.ContainerService/managedClusters/diagnosticLogs/Read", "category":"kube-scheduler", "resourceId":"/SUBSCRIPTIONS/C111111-DXXX-4XXX-AXXX-900000000/RESOURCEGROUPS/AG-AKS-RG/PROVIDERS/MICROSOFT.CONTAINERSERVICE/MANAGEDCLUSTERS/AG-AKS-CLUSTER", "properties":▼{ "log":"W0622 22:50:25.377565 1 reflector.go:256] k8s.io/client-go/informers/factory.go:131: watch of *v1.StorageClass ended with: too old resource version: 3828720 (3970094) ", "stream":"stderr", "pod":"kube-scheduler-744886667c-cxnvk", "containerID":"6093b28d82e1f850fb5a9f59ed8c31aa7179e8a3907449dcd2450a6605341a60" }, "time":"2019-06-22T22:50:25.000Z" }
Query samples
kube-apiserver
_sourceCategory="azure/aks" "kube-apiserver" | json "properties.log", "category", "time", "properties.pod", "resourceId" as log, category, time, pod, resourceId | where category ="kube-apiserver" | parse regex field=log "(?<severity>W|I|F|E)(?<tt>[\S]+) (?<times>[\d:.]+)[\s]+(?<log_msg>.*)" | parse regex field=resourceId "RESOURCEGROUPS\/(?<resource_grp>[\S]+)\/PROVIDERS\/MICROSOFT\.CONTAINERSERVICE\/MANAGEDCLUSTERS\/(?<cluster>[\S]+)" | timeslice 1h | count by _timeslice, severity | transpose row _timeslice column severity | fillmissing timeslice(1h)
kube-controller-manager
_sourceCategory="azure/aks" ("kube-controller-manager") | json "properties.log", "category", "time", "properties.pod", "resourceId" as log, category, time, pod, resourceId | where category ="kube-controller-manager" | parse regex field=log "(?<severity>W|I|F|E)(?<tt>[\S]+) (?<times>[\d:.]+)[\s]+(?<log_msg>.*)" | parse regex field=resourceId "RESOURCEGROUPS\/(?<resource_grp>[\S]+)\/PROVIDERS\/MICROSOFT\.CONTAINERSERVICE\/MANAGEDCLUSTERS\/(?<cluster>[\S]+)" | timeslice 1h | count by _timeslice, severity | transpose row _timeslice column severity | fillmissing timeslice(1h)
kube-scheduler
_sourceCategory="azure/aks" "kube-scheduler" | json "properties.log", "category", "time", "properties.pod", "resourceId" as log, category, time, pod, resourceId | where category ="kube-scheduler" | parse regex field=log "(?<severity>W|I|F|E)(?<tt>[\S]+) (?<times>[\d:.]+)[\s]+(?<log_msg>.*)" | parse regex field=resourceId "RESOURCEGROUPS\/(?<resource_grp>[\S]+)\/PROVIDERS\/MICROSOFT\.CONTAINERSERVICE\/MANAGEDCLUSTERS\/(?<cluster>[\S]+)" | timeslice 1h | count by _timeslice, severity | transpose row _timeslice column severity | fillmissing timeslice(1h)