Skip to main content
Sumo Logic

Collect Logs and Metrics for the Kubernetes Control Plane App

  1. Last updated
  2. Save as PDF

This page provides instructions for configuring log and metric collection for the Kubernetes Control Plane App.

Log and Metric Types

The Kubernetes Control Plane App uses logs and metrics.

Log sources

The Sumo Logic Kubernetes app uses FluentBit and FluentD to collect logs.

Metric sources
  • Kubernetes API Server Metrics.
  • Scheduler Metrics.
  • Controller Manager Metrics. 
  • Node-exporter Metrics. 
  • kube-state-metrics.

Metrics are collected using Prometheus with FluentD. For additional information on metrics options you can configure for collection, see this document.

Configuring log and metric collection 

You set up collection by configuring two apps. The Sumo Logic Kubernetes App provides services for managing and monitoring Kubernetes worker nodes and works in conjunction with the Kubernetes Control Plane App that monitors the master node control plane, including the API server, etcd, kube-system, as well as worker nodes.

Step 1. Set up and install the Kubernetes App

The Sumo Logic Kubernetes App provides the services for managing and monitoring Kubernetes worker nodes. You must set up collection and install the Kubernetes App before you install the Kubernetes - Control Plane App. You configure log and metric collection during this process.

To set up and install the Kubernetes app review our best practices and follow the steps in the Collect Logs and Metrics for the Kubernetes App document.

Step 2. Install the Kubernetes Control Plane App

You configured log and metric collection when you installed the Kubernetes App. You are now ready to install the Kubernetes Control Plane App.

See how to Install the Kubernetes Control Plane App and view the Dashboards.

Sample log messages

Kube API Server Logs
{"timestamp":1562059802772,"log":"E0702 09:30:02.772323       1 
watcher.go:208] watch chan error: etcdserver: mvcc: required revision has been compacted",
"stream":"stdout","time":"2019-07-02T09:30:02.772474301Z"}
Kube Control Manager Logs
 {"timestamp":1561651885393,"log":"E0627 16:11:25.377997       1 
 horizontal.go:214] failed to compute desired number of replicas based on listed metrics for 
 Deployment/2019-06-27-numbers/goledzki-k8sdemo-20190627: Invalid metrics (1 invalid out of 1), 
 last error was: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch 
 metrics from resource metrics API: the server could not find the requested resource 
 (get pods.metrics.k8s.io)","stream":"stderr","time":"2019-06-27T16:11:25.393923292Z"}
Kube Scheduler Logs
 {"timestamp":1561646836232,"log":"E0627 14:47:16.211140       1 
 scheduling_queue.go:346] Unable to find backoff value for pod 
 2019-06-27-numbers/goledzki-k8sdemo-20190627-6cc8d4b677-vqwdt in backoffQ","stream":"stderr",
 "time":"2019-06-27T14:47:16.232275553Z"}
Kube-System Namespace Logs
{"timestamp":1562059802772,"log":"E0702 09:30:02.772323       1 
watcher.go:208] watch chan error: etcdserver: mvcc: required revision has been compacted",
"stream":"stdout","time":"2019-07-02T09:30:02.772474301Z"}
Application Logs
{"timestamp":1561534865020,"log":"E0626 07:41:05.020255       1 
manager.go:101] Error in scraping containers from kubelet:192.168.190.54:10255: 
failed to get all container stats from Kubelet URL \"http://192.168.190.54:10255/stats/container/\": 
Post http://192.168.190.54:10255/stats/container/: dial tcp 192.168.190.54:10255: getsockopt: 
connection refused"}

Query sample

Control Manager - Event Severity Trend
 _sourceCategory = *controller*
| json field=_raw "log" as log
| parse regex field=log "^(?<severity>.)(?:[0-9])"
| timeslice 1h
| count _timeslice, severity
| transpose row _timeslice column severity
| fillmissing timeslice(1h)