Skip to main content
Sumo Logic

Set up collection for Kubernetes

This page provides an overview of the collection process for Kubernetes environments, and then walks you through configuring log and metric collection. The Sumo Logic Kubernetes App provides services for managing and monitoring Kubernetes worker nodes works in conjunction with the Kubernetes Control Plane App that monitors the master node control plane, including the API server, etcd, kube-system, as well as worker nodes. You will set up both of these apps in the configuration process.

Collection overview

Sumo Logic collects logs, events, metrics, and security data with Fluentbit, FluentD, Prometheus, and Falco. These collectors are all open source collectors that are maintained by the cloud native computing foundation (CNCF). The collected data streams through a centralized FluentD pipeline for metadata enrichment. Sumo Logic tags the container, pod, node, and cluster, as well as identifying the service, namespace, and deployment. 

K8s_Centralized_Collection.png

Log and metric types

The Kubernetes Control Plane App uses logs and metrics.

Log sources

The Sumo Logic Kubernetes app uses FluentBit and FluentD to collect logs.

Metric sources
  • Kubernetes API Server Metrics.
  • Scheduler Metrics.
  • Controller Manager Metrics. 
  • Node-exporter Metrics. 
  • kube-state-metrics.

Metrics are collected using Prometheus with FluentD. For additional information on metrics options you can configure for collection, see this document.

Configuring log and metric collection

The Sumo Logic Kubernetes Control Plane App works in conjunction with the  Kubernetes App to monitor the master node control plane, including the API server, etcd, kube-system, as well as worker nodes. You configure log and metric collection when you install the Kubernetes App, as described in this section.

Step 1. Setup and install the Kubernetes App

The Sumo Logic Kubernetes App provides the services for managing and monitoring Kubernetes worker nodes. You must set up collection and install the Kubernetes App before you install the Kubernetes - Control Plane App. You configure log and metric collection during this process.

To set up and install the Kubernetes app, review our best practices and follow the instructions in this document.

Step 2. Install the Kubernetes Control Plane App 

The process for installing the Kubernetes Control Plane App varies depending on the platform of your cluster. This section provides information on the available Kubernetes platforms. Choose the procedure meant for your cluster environment.

Custom Kubernetes cluster

If you built your own Kubernetes cluster, you should follow the steps recommended in this section. You configured log and metric collection when you installed the Kubernetes App, and are now ready to install the Kubernetes Control Plane App.

To install the Kubernetes Control Plane App, follow the instructions on this page.

Managed service provider

If you are using a managed service provider, you should follow the steps recommended in this section for your managed service. You configured log and metric collection when you installed the Kubernetes App, and are now ready to install the appropriate control plane app for your platform:

Best practices

Setting the scrape interval globally in Prometheus

During installation you can set the scrape interval globally for Prometheus to reduce or increase the frequency at which metrics are collected. For example, the following flag passed into the helm install command will set the scrape interval to every 2 minutes instead of every 1 minute (default).

--set prometheus-operator.prometheus.prometheusSpec.scrapeInterval=”2m”

You can also set this in the values.yaml file by adding scrape_interval. See Prometheus documentation for reference.

Considerations:

  • Reducing the scrape interval works best for metrics sources that generate low volumes of metrics.
    • Examples of sources that generate low volumes of metrics
      • kube-controller-manager-metrics
      • kube-scheduler-metrics
    • Examples of sources which generate higher volumes of metrics
      • apiserver-metrics
      • Kube-state-metrics
  • Increasing the scrape interval may affect some dashboard panels, preventing them from rendering properly.

Fluentd collection performance

We have benchmarked Fluentd collection performance of logs and metrics so you can best determine the number of Fluentd replicas needed for your workload. You may use the following observation when sizing your Fluentd deployment.

Using Kubernetes version 1.13 and Sumo Logic Helm chart version 0.6.0-0.8.0 each Fluentd replica could handle the following:

Log volume Metrics data points per minute (DPM)
1.3 MB 0 DPM
750 KB 20K DPM
250 KB 40K DPM
0 bytes 50K DPM

This benchmarking was performed on an AWS EC2 M4.Large instance with 2 vCPU and 8G RAM.

Benchmark Configuration

Fluentd log collection was tested with an internal log generator capable of a production load at variant rates.

Our metrics workload was generated with Avalanche, a load generator for producing Prometheus metrics. Avalanche was configured with the following parameters

--metric-count=200
--series-count=100
--port=9006
--series-interval=60000
--metric-interval=60000
--value-interval=60

Metrics generated by Avalanche were collected by an instance of Prometheus and then forwarded to a single replica of Fluentd, before being sent to a Sumo Logic HTTP Source endpoint.