Skip to main content
Sumo Logic

Collect logs and metrics for the GKE - Control Plane App

This page has instructions for configuring log and metric collection for the Sumo App for GKE - Control Plane.

Collection process overview

After you install the Sumo Logic Kubernetes App, you configure a hosted collector, HTTP source, and establish Kubernetes collection. You will establish the key components for Google Cloud Platform (GCP) services collection which include Google Stackdriver, and Google Cloud Pub/Sub. Finally, you’ll configure metrics collection.

For GCP integration, Google Stackdriver collects logs from GCP services. Once you’ve configured the pipeline, the logs collected by Stackdriver are published to a Google Pub/Sub topic. A Sumo Logic GCP source on a hosted collector subscribed to that topic ingests the logs into Sumo Logic.

GKE-Collection-Process-Overview.png

The configuration process includes the following tasks: 

  1. Install the Sumo Logic Kubernetes App.

  2. Configure a GCP source on a hosted collector, during which you create the HTTP URL for the source. You then use the Google Cloud Console to register the URL as a validated domain.  

  3. Create a topic in Google Pub/Sub and subscribe the GCP source URL to that topic.

  4. Create an export of GCP logs from Stackdriver. This involves writing a filter that selects the log entries to export, then choosing a Pub/Sub as the destination. The filter and destination are held in an object called a sink.

Step 1. Set up and install the Kubernetes App

The Sumo Logic Kubernetes App provides the services for managing and monitoring Kubernetes worker nodes. You must set up collection and  install the Kubernetes App before configuring collection for the GKE App. You will configure log and metric collection during this process.

To set up and install the Kubernetes app, follow the instructions in this document.

Step 2. Configure a Google Cloud Platform Source 

The GCP source receives log data from Google Pub/Sub. The GCP source will only be usable for log data formatted as data coming from Google Pub/Sub.

To configure a Google Platform Source, follow the instructions in this document.

Step 3. Configure a Pub/Sub topic for GCP

Once you configure the Pub/Sub, you can export data from Stackdriver to the Pub/Sub.

To configure a Pub/Sub topic for GCP,  follow the instructions in this document.

Step 4. Create an export of Google Kubernetes Engine logs from Stackdriver

This section walks you through the task of creating an export of Google Kubernetes Engine logs from Stackdriver.

To create an export of GKE logs from Stackdriver, do the following:

  1. Log in to your GCP account and go to Logging and select Exports from the pop-up menu.

GCP_Logging_Exports_dialog.png

  1. Click Create Export.

GKE_Create_Export_option.png

  1. Select GCP services to filter the logs.

GKE now includes managed support for Stackdriver. Installing or updating Stackdriver support happens automatically when you select a GKE version for your cluster and a support option. There are three support options. These options are provided by all GKE versions presently available for new clusters and for updates to existing clusters:

  • Legacy Stackdriver support.
  • Stackdriver Kubernetes Engine Monitoring support.
  • No support for Stackdriver.

GKE_Services.png

Based on the option you choose, the “GCP service” name changes when creating a sink for export as per mapping, as shown in the following table:

  Legacy Stackdriver Stackdriver Kubernetes Engine Monitoring
GKE Cluster Operations Check-mark-symbol-small.png Check-mark-symbol-small.png
Kubernetes Cluster Check-mark-symbol-small.png Check-mark-symbol-small.png
Kubernetes Node   Check-mark-symbol-small.png
Kubernetes Pod   Check-mark-symbol-small.png
  1. Determine the GCP services for your setup to create sinks.

  2. In the Edit Export window on the right do the following:

  1. Enter a Sink Name. For example, "gce-vm-instance".

  2. Select "Cloud Pub/Sub" as the Sink Service.

  3. Set Sink Destination to the Pub/Sub topic you created in the Google Cloud Platform Source procedure. For example, "pub-sub-logs".

  4. Click Create Sink.

  5. Repeat above steps for all the services.

GKE_Edit_Export_options.png 

Sample log messages

StackDriver Monitoring - Container StdErr
{
  "message":{
    "attributes":{
      "logging.googleapis.com/timestamp":"2019-06-26T10:41:00.394447795Z"
    },
    "data":{
      "insertId":"qmuyjp0a5yrmvuyzy",
      "labels":{
        "k8s-pod/app":"prometheus",
        "k8s-pod/controller-revision-hash":"prometheus-prometheus-operator-prometheus-67f77458d4",
        "k8s-pod/prometheus":"prometheus-operator-prometheus",
        "k8s-pod/statefulset_kubernetes_io/pod-name":"prometheus-prometheus-operator-prometheus-0"
      },
   
"logName":"projects/product-sandbox-1/logs/stderr",
      "receiveTimestamp":"2019-06-26T10:41:05.973133299Z",
      "resource":{
        "labels":{
          "cluster_name":"arun-gke-stackdriver-engine-monitoring",
          "container_name":"prometheus",
          "location":"us-central1-a",
          "namespace_name":"sumologic",
          "pod_name":"prometheus-prometheus-operator-prometheus-0",
          "project_id":"product-sandbox-1"
        },
        "type":"k8s_container"
      },
      "severity":"ERROR",
      "textPayload":"ts=2019-06-26T10:41:00.394Z caller=dedupe.go:111 component=remote level=info queue=13:http://fluentd:9888/prometheus.metrics.apiserver msg=\"Remote storage resharding\" from=6 to=3\n",
      "timestamp":"2019-06-26T10:41:00.394447795Z"
    },
    "messageId":"597120419207003",
    "message_id":"597120419207003",
    "publishTime":"2019-06-26T10:41:06.382Z",
    "publish_time":"2019-06-26T10:41:06.382Z"
  },
  
"subscription":"projects/product-sandbox-1/subscriptions/sumo_gke_stackdriver"
}   
StackDriver Monitoring - Container StdOut
 {
  "message":{
    "attributes":{
      "logging.googleapis.com/timestamp":"2019-06-26T10:44:04.40824208Z"
    },
    "data":{
      "insertId":"hz9rs3tndyceb4rvh",
      "labels":{
        "k8s-pod/component":"test-logger",
        "k8s-pod/pod-template-hash":"7896bcb5cd"
      },
      "logName":"projects/product-sandbox-1/logs/stdout",
      "receiveTimestamp":"2019-06-26T10:44:06.456680433Z",
      "resource":{
        "labels":{
          "cluster_name":"arun-gke-stackdriver-engine-monitoring",
          "container_name":"test-logger",
          "location":"us-central1-a",
          "namespace_name":"default",
          "pod_name":"test-logger-7896bcb5cd-954mz",
          "project_id":"product-sandbox-1"
        },
        "type":"k8s_container"
      },
      "severity":"INFO",
      "textPayload":"Processing credit card 1234 5678 9012 3456\n",
      "timestamp":"2019-06-26T10:44:04.40824208Z"
    },
    "messageId":"597120030247162",
    "message_id":"597120030247162",
    "publishTime":"2019-06-26T10:44:06.965Z",
    "publish_time":"2019-06-26T10:44:06.965Z"
  },
  "subscription":"projects/product-sandbox-1/subscriptions/sumo_gke_stackdriver"
}
StackDriver Monitoring - Events
 
{
  "message":{
"attributes":{
   "logging.googleapis.com/timestamp":"2019-06-26T10:38:14Z"
},
"data":{
   "insertId":"1o05pug1iym1ef",
   "jsonPayload":{
     "apiVersion":"v1",
     "involvedObject":{
       "apiVersion":"v1",
       "kind":"Pod",
       "name":"mysql-fc99db7b8-jv7tl",
       "namespace":"robot-shop",
       "resourceVersion":"5625122",
       "uid":"5548f14c-8e79-11e9-bbfd-42010a8002a8"
     },
     "kind":"Event",
     "message":"0/3 nodes are available: 3 Insufficient cpu.",
     "metadata":{
       "creationTimestamp":"2019-06-23T21:57:27Z",
       "name":"mysql-fc99db7b8-jv7tl.15aaf2ad63dddd2c",
       "namespace":"robot-shop",
       "resourceVersion":"50268",
       "selfLink":"/api/v1/namespaces/robot-shop/events/mysql-fc99db7b8-jv7tl.15aaf2ad63dddd2c",
       "uid":"e3de27c7-9601-11e9-870d-42010a80023c"
     },
     "reason":"FailedScheduling",
     "source":{
       "component":"default-scheduler"
     },
     "type":"Warning"
   },
   "logName":"projects/product-sandbox-1/logs/events",
   "receiveTimestamp":"2019-06-26T10:38:19.792891713Z",
   "resource":{
     "labels":{
       "cluster_name":"arun-gke-stackdriver-engine-monitoring",
       "location":"us-central1-a",
       "namespace_name":"robot-shop",
       "pod_name":"mysql-fc99db7b8-jv7tl",
       
"project_id":"product-sandbox-1"
     },
     "type":"k8s_pod"
   },
   "severity":"WARNING",
   "timestamp":"2019-06-26T10:38:14Z"
},
"messageId":"597107305873932",
"message_id":"597107305873932",
"publishTime":"2019-06-26T10:38:20.474Z",
"publish_time":"2019-06-26T10:38:20.474Z"
  },
  "subscription":"projects/product-sandbox-1/subscriptions/sumo_gke_stackdriver"
}
Legacy StackDriver - Container StdErr
{
  "message":{
    "attributes":{
      "logging.googleapis.com/timestamp":"2019-06-26T10:47:13.128751259Z"
    },
    "data":{
      "insertId":"1cakep6g1ncgxlq",
      "labels":{
        "compute.googleapis.com/resource_name":"fluentd-gcp-v3.2.0-5j8t8",
        "container.googleapis.com/namespace_name":"sumologic",
        "container.googleapis.com/pod_name":"prometheus-prometheus-operator-prometheus-0",
        "container.googleapis.com/stream":"stderr"
      },
      "logName":"projects/product-sandbox-1/logs/prometheus",
      "receiveTimestamp":"2019-06-26T10:47:19.525038772Z",
      "resource":{
        "labels":{
          "cluster_name":"arun-gke-cluster",
          "container_name":"prometheus",
          "instance_id":"3947999507639860837",
          "namespace_id":"sumologic",
          "pod_id":"prometheus-prometheus-operator-prometheus-0",
          "project_id":"product-sandbox-1",
          "zone":"us-central1-a" 
          
},
        "type":"container"
      },
      "severity":"ERROR",
      "textPayload":"ts=2019-06-26T10:47:13.128Z caller=dedupe.go:111 component=remote level=info queue=19:http://fluentd:9888/prometheus.metrics.container msg=\"Remote storage resharding\" from=373 to=92\n",
      "timestamp":"2019-06-26T10:47:13.128751259Z"
    },
    "messageId":"597119793276049",
    "message_id":"597119793276049",
    "publishTime":"2019-06-26T10:47:20.275Z",
    "publish_time":"2019-06-26T10:47:20.275Z"
  },
  "subscription":"projects/product-sandbox-1/subscriptions/gke_cluster"
}
Legacy StackDriver - Container StdOut
{
  "message":{
    "attributes":{
      "logging.googleapis.com/timestamp":"2019-06-26T10:47:36.932520043Z"
    },
    "data":{
      "insertId":"qczw0afzu9akl",
      "labels":{
        "compute.googleapis.com/resource_name":"fluentd-gcp-v3.2.0-vkzt5",
        "container.googleapis.com/namespace_name":"default",
        "container.googleapis.com/pod_name":"test-logger-76d458db56-gl4fq",
        "container.googleapis.com/stream":"stdout"
      },
      "logName":"projects/product-sandbox-1/logs/test-logger",
      "receiveTimestamp":"2019-06-26T10:47:40.192057183Z",
      "resource":{
        "labels":{
          "cluster_name":"arun-gke-cluster",
          
"container_name":"test-logger",
          "instance_id":"8083082308129291877",
          "namespace_id":"default",
          "pod_id":"test-logger-76d458db56-gl4fq",
          "project_id":"product-sandbox-1",
          "zone":"us-central1-a"
        },
        "type":"container"
      },
      "severity":"INFO",
      "textPayload":"Processing credit card 1234 5678 9012 3456\n",
      "timestamp":"2019-06-26T10:47:36.932520043Z"
    },
    "messageId":"597121135964578",
    "message_id":"597121135964578",
    "publishTime":"2019-06-26T10:47:40.423Z",
    "publish_time":"2019-06-26T10:47:40.423Z"
  },
  "subscription":"projects/product-sandbox-1/subscriptions/gke_cluster"
}
Legacy StackDriver Monitoring - Events
 {
  "message":{
    "attributes":{
      "logging.googleapis.com/timestamp":"2019-06-26T10:45:28Z"
    },
    "data":{
      "insertId":"12l15ytg1iuthzk",
      "jsonPayload":{
        "apiVersion":"v1",
        "involvedObject":{
          "apiVersion":"v1",
          "kind":"Pod",
          "name":"fluentd-594f98d6c8-jzpfs",
          "namespace":"sumologic",
          "resourceVersion":"1535338",
          "uid":"f8a6c405-8395-11e9-9642-42010a800109"
        },
        "kind":"Event",
        "message":"MountVolume.SetUp failed for volume \"fluentd-token-sr2xf\" : secrets \"fluentd-token-sr2xf\" not found",
        "metadata":{
          "creationTimestamp":"2019-06-04T09:53:35Z",
          
"name":"fluentd-594f98d6c8-jzpfs.15a4f62662893b2f",
          "namespace":"sumologic",
          "resourceVersion":"132802",
          "selfLink":"/api/v1/namespaces/sumologic/events/fluentd-594f98d6c8-jzpfs.15a4f62662893b2f",
          "uid":"9ebb2fed-86ae-11e9-bcca-42010a800109"
        },
        "reason":"FailedMount",
        "source":{
          "component":"kubelet",
          "host":"gke-arun-gke-cluster-default-pool-4b292970-g7qt"
        },
        "type":"Warning"
      },
      "logName":"projects/product-sandbox-1/logs/events",
      "receiveTimestamp":"2019-06-26T10:45:33.816643511Z",
      "resource":{
        "labels":{
          "cluster_name":"arun-gke-cluster",
          "location":"us-central1-a",
          "project_id":"product-sandbox-1"
        },
        "type":"gke_cluster"
      },
      "severity":"WARNING",
      "timestamp":"2019-06-26T10:45:28Z"
    },
    "messageId":"597115129096918",
    "message_id":"597115129096918",
    "publishTime":"2019-06-26T10:45:34.192Z",
    "publish_time":"2019-06-26T10:45:34.192Z"
  },
  "subscription":"projects/product-sandbox-1/subscriptions/gke_cluster"
}

Query Sample

 Error Stream - Stackdriver Monitoring
_source="GKE Cloud Logs - Stackdriver" error
| parse regex "\"logName\":\"(?<log_name>[^\"]+)\""
| json field=_raw "message.data.jsonPayload.message" as message
| json "message.data.resource.labels" as labels
| json field=labels "project_id", "cluster_name" as project, cluster
| json field=_raw "message.data.timestamp" as timestamp
| count by timestamp, project, cluster,log_name, message
Created Resources by Node Over Time - Legacy Stackdriver
_sourceCategory = "GKE Cloud Logs - Legacy Stackdriver" logName reason host "\"type\":\"gke_cluster\"" "\"reason\":\"Created\""
| parse regex "\"logName\":\"(?<log_name>[^\"]+)\""
| where log_name matches "projects/*/logs/events"
| json "message.data.resource.labels", "message.data.jsonPayload.source.host" as labels, node
| json field=labels "project_id", "cluster_name" as project, cluster
| timeslice 1h
| count as eventCount by _timeslice, node, cluster, project
| transpose row _timeslice column node, cluster, project
| fillmissing timeslice(1h)