Skip to main content
Sumo Logic

Collect Logs and Metrics for the Kubernetes Control Plane App

  1. Last updated
  2. Save as PDF

This page provides instructions for configuring log and metric collection for the Kubernetes Control Plane App.

Log and Metric Types

The Kubernetes Control Plane App uses logs and metrics.

Log sources

The Sumo Logic Kubernetes app uses FluentBit and FluentD to collect logs.

Metric sources
  • Kubernetes API Server Metrics.
  • Scheduler Metrics.
  • Controller Manager Metrics. 
  • Node-exporter Metrics. 
  • kube-state-metrics.

For more information, see this page. Metrics are collected using Prometheus with FluentD.

Configuring log and metric collection 

The Sumo Logic Kubernetes Control Plane App works in conjunction with the  Kubernetes App to monitor the master node control plane, including the API server, etcd, kube-system, as well as worker nodes. You configure log and metric collection when you install the Kubernetes App, as described in this section.

Step 1. Set up and install the Kubernetes App

The Sumo Logic Kubernetes App provides the services for managing and monitoring Kubernetes worker nodes. You must set up collection and  install the Kubernetes App before configuring collection for the Kubernetes - Control Plane App. You will configure log and metric collection during this process.

To set up and install the Kubernetes app, follow the instructions in this document.

Step 2. Install the Kubernetes Control Plane App

You configured log and metric collection during the Kubernetes App installation. You are now ready to install the Kubernetes Control Plane App.

To install the Kubernetes Control Plane App, follow the instruction on this page.

Sample log messages

Kube API Server Logs
{"timestamp":1562059802772,"log":"E0702 09:30:02.772323       1 
watcher.go:208] watch chan error: etcdserver: mvcc: required revision has been compacted",
"stream":"stdout","time":"2019-07-02T09:30:02.772474301Z"}
Kube Control Manager Logs
 {"timestamp":1561651885393,"log":"E0627 16:11:25.377997       1 
 horizontal.go:214] failed to compute desired number of replicas based on listed metrics for 
 Deployment/2019-06-27-numbers/goledzki-k8sdemo-20190627: Invalid metrics (1 invalid out of 1), 
 last error was: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch 
 metrics from resource metrics API: the server could not find the requested resource 
 (get pods.metrics.k8s.io)","stream":"stderr","time":"2019-06-27T16:11:25.393923292Z"}
Kube Scheduler Logs
 {"timestamp":1561646836232,"log":"E0627 14:47:16.211140       1 
 scheduling_queue.go:346] Unable to find backoff value for pod 
 2019-06-27-numbers/goledzki-k8sdemo-20190627-6cc8d4b677-vqwdt in backoffQ","stream":"stderr",
 "time":"2019-06-27T14:47:16.232275553Z"}
Kube-System Namespace Logs
{"timestamp":1562059802772,"log":"E0702 09:30:02.772323       1 
watcher.go:208] watch chan error: etcdserver: mvcc: required revision has been compacted",
"stream":"stdout","time":"2019-07-02T09:30:02.772474301Z"}
Application Logs
{"timestamp":1561534865020,"log":"E0626 07:41:05.020255       1 
manager.go:101] Error in scraping containers from kubelet:192.168.190.54:10255: 
failed to get all container stats from Kubelet URL \"http://192.168.190.54:10255/stats/container/\": 
Post http://192.168.190.54:10255/stats/container/: dial tcp 192.168.190.54:10255: getsockopt: 
connection refused"}

Query sample

Control Manager - Event Severity Trend
 _sourceCategory = *controller*
| json field=_raw "log" as log
| parse regex field=log "^(?<severity>.)(?:[0-9])"
| timeslice 1h
| count _timeslice, severity
| transpose row _timeslice column severity
| fillmissing timeslice(1h)