Skip to main content
Sumo Logic

Collect Logs and Metrics for the Kubernetes Control Plane App

  1. Last updated
  2. Save as PDF

This page provides instructions for configuring log and metric collection for the Kubernetes Control Plane App.

Log and Metric Types

The Kubernetes Control Plane App uses logs and metrics.

Log sources

The Sumo Logic Kubernetes app uses FluentBit and FluentD to collect logs.

Metric sources
  • Kubernetes API Server Metrics.
  • Scheduler Metrics.
  • Controller Manager Metrics. 
  • Node-exporter Metrics. 
  • kube-state-metrics.

Metrics are collected using Prometheus with FluentD. For additional information on metrics options you can configure for collection, see this document.

Configuring log and metric collection 

Reference the Deployment Guide in our sumologic-kubernetes-collection GitHub repository for detailed instructions on how to collect Kubernetes logs, metrics, and events; enrich them with deployment, pod, and service level metadata; and send them to Sumo Logic.

The Deployment Guide has information on advanced configurations, best practices, performance, troubleshooting, and upgrading for our latest and previous versions of supported software.

Apps

The Sumo Logic Kubernetes App provides the services for managing and monitoring Kubernetes worker nodes. You must set up collection and install the Kubernetes App before you install the Kubernetes - Control Plane App.

After the Kubernetes App is installed you can Install the Kubernetes Control Plane App and view the Dashboards.

Sample log messages

Kube API Server Logs
{"timestamp":1562059802772,"log":"E0702 09:30:02.772323       1 
watcher.go:208] watch chan error: etcdserver: mvcc: required revision has been compacted",
"stream":"stdout","time":"2019-07-02T09:30:02.772474301Z"}
Kube Control Manager Logs
 {"timestamp":1561651885393,"log":"E0627 16:11:25.377997       1 
 horizontal.go:214] failed to compute desired number of replicas based on listed metrics for 
 Deployment/2019-06-27-numbers/goledzki-k8sdemo-20190627: Invalid metrics (1 invalid out of 1), 
 last error was: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch 
 metrics from resource metrics API: the server could not find the requested resource 
 (get pods.metrics.k8s.io)","stream":"stderr","time":"2019-06-27T16:11:25.393923292Z"}
Kube Scheduler Logs
 {"timestamp":1561646836232,"log":"E0627 14:47:16.211140       1 
 scheduling_queue.go:346] Unable to find backoff value for pod 
 2019-06-27-numbers/goledzki-k8sdemo-20190627-6cc8d4b677-vqwdt in backoffQ","stream":"stderr",
 "time":"2019-06-27T14:47:16.232275553Z"}
Kube-System Namespace Logs
{"timestamp":1562059802772,"log":"E0702 09:30:02.772323       1 
watcher.go:208] watch chan error: etcdserver: mvcc: required revision has been compacted",
"stream":"stdout","time":"2019-07-02T09:30:02.772474301Z"}
Application Logs
{"timestamp":1561534865020,"log":"E0626 07:41:05.020255       1 
manager.go:101] Error in scraping containers from kubelet:192.168.190.54:10255: 
failed to get all container stats from Kubelet URL \"http://192.168.190.54:10255/stats/container/\": 
Post http://192.168.190.54:10255/stats/container/: dial tcp 192.168.190.54:10255: getsockopt: 
connection refused"}

Query sample

Control Manager - Event Severity Trend
 _sourceCategory = *controller*
| json field=_raw "log" as log
| parse regex field=log "^(?<severity>.)(?:[0-9])"
| timeslice 1h
| count _timeslice, severity
| transpose row _timeslice column severity
| fillmissing timeslice(1h)