Collect Logs and Metrics for the Kubernetes Control Plane App
This page provides instructions for configuring log and metric collection for the Kubernetes Control Plane App.
Log and Metric Types
The Kubernetes Control Plane App uses logs and metrics.
Log sources
The Sumo Logic Kubernetes app uses FluentBit and FluentD to collect logs.
Metric sources
- Kubernetes API Server Metrics.
- Scheduler Metrics.
- Controller Manager Metrics.
- Node-exporter Metrics.
- kube-state-metrics.
Metrics are collected using Prometheus with FluentD. For additional information on metrics options you can configure for collection, see this document.
Configuring log and metric collection
The Sumo Logic Kubernetes Control Plane App works in conjunction with the Kubernetes App to monitor the master node control plane, including the API server, etcd, kube-system, as well as worker nodes. You configure log and metric collection when you install the Kubernetes App, as described in this section.
Step 1. Set up and install the Kubernetes App
The Sumo Logic Kubernetes App provides the services for managing and monitoring Kubernetes worker nodes. You must set up collection and install the Kubernetes App before you install the Kubernetes - Control Plane App. You configure log and metric collection during this process.
To set up and install the Kubernetes app, review our best practices and follow the instructions in this document.
Step 2. Install the Kubernetes Control Plane App
You configured log and metric collection when you installed the Kubernetes App. You are now ready to install the Kubernetes Control Plane App.
To install the Kubernetes Control Plane App, follow the instruction on this page.
Sample log messages
Kube API Server Logs
{"timestamp":1562059802772,"log":"E0702 09:30:02.772323 1
watcher.go:208] watch chan error: etcdserver: mvcc: required revision has been compacted",
"stream":"stdout","time":"2019-07-02T09:30:02.772474301Z"}
Kube Control Manager Logs
{"timestamp":1561651885393,"log":"E0627 16:11:25.377997 1
horizontal.go:214] failed to compute desired number of replicas based on listed metrics for
Deployment/2019-06-27-numbers/goledzki-k8sdemo-20190627: Invalid metrics (1 invalid out of 1),
last error was: failed to get cpu utilization: unable to get metrics for resource cpu: unable to fetch
metrics from resource metrics API: the server could not find the requested resource
(get pods.metrics.k8s.io)","stream":"stderr","time":"2019-06-27T16:11:25.393923292Z"}
Kube Scheduler Logs
{"timestamp":1561646836232,"log":"E0627 14:47:16.211140 1
scheduling_queue.go:346] Unable to find backoff value for pod
2019-06-27-numbers/goledzki-k8sdemo-20190627-6cc8d4b677-vqwdt in backoffQ","stream":"stderr",
"time":"2019-06-27T14:47:16.232275553Z"}
Kube-System Namespace Logs
{"timestamp":1562059802772,"log":"E0702 09:30:02.772323 1
watcher.go:208] watch chan error: etcdserver: mvcc: required revision has been compacted",
"stream":"stdout","time":"2019-07-02T09:30:02.772474301Z"}
Application Logs
{"timestamp":1561534865020,"log":"E0626 07:41:05.020255 1
manager.go:101] Error in scraping containers from kubelet:192.168.190.54:10255:
failed to get all container stats from Kubelet URL \"http://192.168.190.54:10255/stats/container/\":
Post http://192.168.190.54:10255/stats/container/: dial tcp 192.168.190.54:10255: getsockopt:
connection refused"}
Query sample
Control Manager - Event Severity Trend
_sourceCategory = *controller* | json field=_raw "log" as log | parse regex field=log "^(?<severity>.)(?:[0-9])" | timeslice 1h | count _timeslice, severity | transpose row _timeslice column severity | fillmissing timeslice(1h)