Collect Logs and Metrics for the Kubernetes Control Plane App
This page provides instructions for configuring log and metric collection for the Kubernetes Control Plane App.
Log and Metric Types
The Kubernetes Control Plane App uses logs and metrics.
Log sources
The Sumo Logic Kubernetes app uses FluentBit and FluentD to collect logs.
Metric sources
- Kubernetes API Server Metrics.
- Scheduler Metrics.
- Controller Manager Metrics.
- Node-exporter Metrics.
- kube-state-metrics.
Metrics are collected using Prometheus with FluentD. For additional information on metrics options you can configure for collection, see this document.
Configuring log and metric collection
Apps
The Sumo Logic Kubernetes App provides the services for managing and monitoring Kubernetes worker nodes. You must set up collection and install the Kubernetes App before you install the Kubernetes - Control Plane App.
After the Kubernetes App is installed you can Install the Kubernetes Control Plane App and view the Dashboards.
Sample log messages
Kube API Server Logs
{"timestamp":1562059802772,"log":"E0702 09:30:02.772323 1 watcher.go:208] watch chan error: etcdserver: mvcc: required revision has been compacted", "stream":"stdout","time":"2019-07-02T09:30:02.772474301Z"}
Kube Control Manager Logs
{"timestamp":1561651885393,"log":"E0627 16:11:25.377997 1 horizontal.go:214] failed to compute desired number of replicas based on listed metrics for Deployment/2019-06-27-numbers/goledzki-k8sdemo-20190627: Invalid metrics (1 invalid out of 1), last error was: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)","stream":"stderr","time":"2019-06-27T16:11:25.393923292Z"}
Kube Scheduler Logs
{"timestamp":1561646836232,"log":"E0627 14:47:16.211140 1 scheduling_queue.go:346] Unable to find backoff value for pod 2019-06-27-numbers/goledzki-k8sdemo-20190627-6cc8d4b677-vqwdt in backoffQ","stream":"stderr", "time":"2019-06-27T14:47:16.232275553Z"}
Kube-System Namespace Logs
{"timestamp":1562059802772,"log":"E0702 09:30:02.772323 1 watcher.go:208] watch chan error: etcdserver: mvcc: required revision has been compacted", "stream":"stdout","time":"2019-07-02T09:30:02.772474301Z"}
Application Logs
{"timestamp":1561534865020,"log":"E0626 07:41:05.020255 1 manager.go:101] Error in scraping containers from kubelet:192.168.190.54:10255: failed to get all container stats from Kubelet URL \"http://192.168.190.54:10255/stats/container/\": Post http://192.168.190.54:10255/stats/container/: dial tcp 192.168.190.54:10255: getsockopt: connection refused"}
Query sample
Control Manager - Event Severity Trend
_sourceCategory = *controller* | json field=_raw "log" as log | parse regex field=log "^(?<severity>.)(?:[0-9])" | timeslice 1h | count _timeslice, severity | transpose row _timeslice column severity | fillmissing timeslice(1h)