Skip to main content
Sumo Logic

Collect Logs for Kubernetes

How to set up log collection for Kubernetes.

Step 1. Create hosted collector and HTTP source

In this step you create, on the Sumo service, an HTTP endpoint to receive your logs. This process involves creating an HTTP source on a hosted collector in Sumo. In Sumo, collectors use sources to receive data.

To create a hosted collector and an HTTP source

  1. If you don’t already have a Sumo account, you can create one by clicking the Free Trial button on https://www.sumologic.com/.

  2. Create a hosted collector, following the instructions on Configure a Hosted Collector in Sumo help. (If you already have a Sumo hosted collector that you want to use, skip this step.)

  3. Create an HTTP source on the collector you created in the previous step. For instructions, see HTTP Logs and Metrics Source in Sumo help.

  4. When you have configured the HTTP source, Sumo will display the URL of the HTTP endpoint. Make a note of the URL. You will use it when you configure the Kubernetes service to send data to Sumo.

Step 2. Create a Kubernetes secret

Create a secret in Kubernetes with the HTTP source URL. If you want to change the secret name, you must modify the Kubernetes manifest accordingly.

kubectl create secret generic sumologic --from-literal=collector-url=<HTTP_Soucre_URL>

Where <HTTP_Source_URL> is the URL displayed when you created the HTTP source, in Step 1 Create hosted collector and HTTP source.

You should see this confirmation message:

secret "sumologic" created

Step 3. Install the Sumo Kubernetes FluentD plugin

In this step, you install the FluendD plugin using kubectl.

Before you start, see Environment variables in the FluendD readme on Github for information about settings you can customize, and how to use annotations to override selected environment variables and exclude data from being sent to Sumo. Environment variables allow you to provide custom metadata values.

  1. Clone the GitHub repo.

  2. The repo contains two different versions of fluentd.yaml. The one you use depends on whether or not you use RBAC for authorization. RBAC is enabled by default as of Kubernetes 1.6.

    If you do not use RBAC (Kubernetes 1.5 and below), run this command:

    kubectl create -f /daemonset/nonrbac/fluentd.yaml

    If you use RBAC (Kubernetes 1.6 and above), run this command:

    kubectl create -f /daemonset/rbac/fluentd.yaml

The following logs should begin flowing into Sumo within a few minutes of plugin installation:

  • Kube API Server Logs
  • Kube Control Manager Logs
  • Kube Scheduler Logs
  • Kube-System Namespace Logs
  • Container Logs

Step 4. Collect Kubernetes API Logs

In this step, you configure the collection of Kubernetes API logs.  We provide a Dockerized python script to extract information from the Kubernetes API and forward to Sumo.

To collect  Kubernetes API  Logs

  1. Create an HTTP source on the collector you created in Step 1 Create hosted collector and HTTP source. For instructions, see HTTP Logs and Metrics Source in Sumo help. Set the Source Category for the source to k8s/api/

  2. When you have configured the HTTP source, Sumo will display the URL of the HTTP endpoint. Make a note of the URL.

  3. Clone the sumologic-k8s-api repository, which contains the Dockerized python script.

  4. If you are using RBAC in your Kubernetes cluster, open sumologic-k8s-api-cronjob-rbac.yaml. If you are not using RBAC, then open sumologic-k8s-api-cronjob.yaml

  5. Insert URL for the HTTP source in the env section of the YAML file.

  6. (Optional) Update the CronJob Schedule if needed. By default it runs every 5 minutes.

  7. Run one of the following commands, depending on whether or not you use RBAC:

    If you use RBAC, run this command:

    kubectl create -f sumologic-k8s-api-cronjob-rbac.yaml


    If you do not use RBAC, run this command:

    kubectl create -f sumologic-k8s-api-cronjob.yaml

Collect Metrics for Kubernetes with Heapster

The recommended way to collect metrics from Kubernetes clusters is to use Heapster and a Sumo collector with a Graphite source. Heapster aggregates metrics across a Kubernetes cluster. Heapster runs as a pod in the cluster, and discovers all nodes in the cluster and queries usage information from each node's kubelet—the on-machine Kubernetes agent. Heapster provides metrics at the cluster, node and pod level.

In this step, you will use YAML configuration files to install or configure the following Kubernetes resources:

  • Heapster—The heapster.yaml file installs Heapster in your Kubernetes cluster and configure a Graphite Sink to send metrics data in Graphite format to the Sumo Collector, which will run as a deployment in your Kubernetes cluster.

  • Config Map—The configmap.yaml file provides configuration settings to the deployment.yaml file, described below, which deploys a Sumo Installed Collector to your Kubernetes cluster.

  • Kubenetes service—Kubernetes Pods use Kubernetes services to talk to each other. The service.yaml creates a service that Heapster will use to communicate with the Sumo Collector Pod which is running as a deployment.

  • Deployment—deployment.yaml deploys two replicas of the of the Sumo Installed Collector (each named “sumo-graphite”) in one of the nodes of your Kubernetes cluster.  This Kubernetes deployment uses the sumo-sources config map (which is created by configmap.yaml file) to configure the Sumo Installed Collector in your Kubernetes cluster.

kubernetes-metrics.png

Apply the YAML configurations

Apply the YAML configurations with following commands

Kubectl apply -f heapster.yaml
Kubectl apply -f configmap.yaml
Kubectl apply -f service.yaml
Kubectl apply -f deployment.yaml

Verify metric collection

Run following commands to make sure everything is running:

kubectl get service

kubectl describe service sumo-graphite

kubectl get pods

kubectl get all

After a few minutes, log in to the Sumo web app, and go to Collection page (Manage Data > Collection > Collection) to verify that the Sumo Collector is receiving metrics from Heapster.

YAML configuration files

This section contains the YAML files described in Collect Metrics for Kubernetes via Heapster.

heapster.yaml

This section contains the contents of heapster.yaml.

The Graphite sink is configured to send metrics to the “sumo-graphite” service with the default namespace.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: heapster
  namespace: kube-system
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: heapster
  namespace: kube-system
spec:
  replicas: 1
  template:
    metadata:
      labels:
        task: monitoring
        k8s-app: heapster
    spec:
      serviceAccountName: heapster
      containers:
      - name: heapster
        image: k8s.gcr.io/heapster-amd64:v1.4.2
        imagePullPolicy: IfNotPresent
        command:
        - /heapster
        - --source=kubernetes:https://kubernetes.default
        - --sink=influxdb:http://monitoring-influxdb.kube-system.svc:8086
        - --sink=graphite:tcp://sumo-graphite.default:2003?prefix=kubernetes.training

configmap.yaml

kind: ConfigMap
apiVersion: v1
metadata:
  name: "sumo-sources"
data:
  sources.json: |-
    {
      "api.version": "v1",
      "sources": [
        {
          "name": "Ankit Test",
          "category": "kubernetes/metrics",
          "automaticDateParsing": true,
          "contentType": "Graphite",
          "timeZone": "UTC",
          "encoding": "UTF-8",
          "protocol": "TCP",
          "port": 2003,
          "sourceType": "Graphite"
        }
      ]
    }

deployment.yaml

Substitute your Sumo Logic Access ID and Key for the placeholders in the env:value tag.

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  labels:
    app: sumo-graphite
  name: sumo-graphite
spec:
  replicas: 2
  template:
    metadata:
      labels:
        app: sumo-graphite
    spec:
      volumes:
      - name: sumo-sources
        configMap:
          name: sumo-sources
          items:
          - key: sources.json
            path: sources.json
      containers:
      - name: sumo-graphite
        image: sumologic/collector:latest
        ports:
        - containerPort: 2003
        volumeMounts:
        - mountPath: /sumo
          name: sumo-sources
        env:
        - name: SUMO_ACCESS_ID
          value: <SUMO_ACCESS_ID> 
        - name: SUMO_ACCESS_KEY
          value: <SUMO_ACCESS_KEY>
        - name: SUMO_SOURCES_JSON
          value: /sumo/sources.json

service.yaml

The service.yaml file, excerpted below, creates a service named “sumo-graphite” (this is the host of Graphite/Carbon sink in your heapster configuration), which will allow communication with pods with app name sumo-graphite (referred in service.yaml as selector tag).

apiVersion: v1
kind: Service
metadata:
  name: sumo-graphite
spec:
  ports:
    - port: 2003
  selector:
    app: sumo-graphite