Skip to main content
Sumo Logic

Lab 2 - Using Explore to Manage Kubernetes Data

Use the Explore view which contains pre-built dashboards that allow you traverse the hierarchy and do cross signal correlation as you monitor, analyze and troubleshoot the Kubernetes environment.

Getting Started

Sumo Logic unifies metadata across the Kubernetes system, providing consistent metadata between logs, messages, events, and security. Metadata enrichment, such as pods, containers, namespaces, deployments, and services, allows users to do cross signal correlation in monitoring, analysis, and troubleshooting - users can move from an event, to a log, to a metric to get the full picture. 

In this lab, you will learn to use the Explore view which contains pre-built dashboards that allow you traverse the hierarchy and do cross signal correlation as you monitor, analyze and troubleshoot the Kubernetes environment.

Unified and consistent Kubernetes metadata between logs, messages, events and security 

One of the really powerful features is that we can unify the metadata across the Kubernetes system and provide consistent metadata between logs, messages, events, and security. Sumo Logic pumps all collected data through fluentd for metadata enrichment. Kubernetes is rich with metadata, to name a few examples - pods, containers, namespaces, deployments, and services. This allows you to do cross signal correlation both in monitoring, analysis, and troubleshooting. Because of this, you can move from an event, to a log, to a metric because they are all being emitted from the same place, see the following diagram.

Screen Shot 2019-09-04 at 11.10.47 PM.png

In this lab, you take a look at a Kubernetes dashboards viewed through our Explore tab. You will learn how to navigate and observe your consistent metadata from the different views. 

  1. Open an Explore tab. On the Home page, click +New.

    NewExplore.png

  2. Click Explore (you can also use Alt+g).

  3. In Explore By, make sure Kubernetes Service View is selected. If needed, select Kubernetes Service View.

    Screen Shot 2019-09-05 at 5.31.29 PM.png

  4. What displays is our Kubernetes Service View, with the Kubernetes - API Server dashboard.  The API is the gateway into the master node of the Kubernetes system. This displays information on the API server logs, which is the control plane component that exposes the Kubernetes API. The panels show details on the API server errors, warnings, and activities.

    Using this view, we can monitor the health and performance of the API server, or review any of the following: server request rates, server success/failure request rates, client activity, and server errors for troubleshooting insights.

  5. To drill down into the hierarchy of the service view, click prod01.travellogic.info.
    Screen Shot 2019-09-04 at 11.13.00 PM.png

    Before we continue to navigate down into our hierarchy, you may want to know how to go up in the hierarchy.

  6. To go back up in the hierarchy, click prod01.travellogic.info. Our user interface toggles between expanding down into the hierarchy or going back up with a single click.
    Screen Shot 2019-09-04 at 11.16.54 PM.png

  7. We are at the cluster level, and we want to continue to go down into our hierarchy of the service view, click prod01.travellogic.info, again.

    You will see the next level in the hierarchy called namespace.

    Screen Shot 2019-09-04 at 10.33.29 PM.png
    A namespace provides a scope for names. Names of resources need to be unique within a namespace, but not across namespaces. It is useful for when you need to divide cluster resources between multiple users.

  8. To navigate further into the service view hierarchy, under prod01.travellogic.info cluster, click sumologic.

  9. Since we are in the Kubernetes Service View, all the services under the prod01.travellogic.info cluster, and sumologic namespace are shown.

    Screen Shot 2019-09-05 at 5.42.34 PM.png

  10. Look at the panels and if you see No Data To Display it means that there is no data based on the filter conditions. This maybe because there are no errors within that time range, as in the top panel titled Errors by Container. 

  11. Let’s go and examine this more carefully. To activate the panel, click in the middle of the Errors by Container panel.

    Screen Shot 2019-09-04 at 10.39.02 PM.png

  12. At the top right of the panel, you will see the time range. It is -15m, which defines the range to be the last 15 minutes of data. Let’s widen the range to the last 24 hours to see if we had any Errors by Container. Choose -15m.

  13. From the Relative time options, select Last 24 Hours.

    Screen Shot 2019-09-04 at 10.41.19 PM.png


    You will now see a pie chart, displaying Errors by Container. It shows you what container produced errors in the last 24 hours in the sumologic namespace . What you just learned is that although no errors were reported in the last 15 minutes, there were errors in the last day. This is easily shown by changing the panel time range filter. Now, if we had received the No Data To Display again, then that tells us there were no errors in the new time range we defined.

    Screen Shot 2019-09-04 at 10.43.04 PM.png

  14. Now that we traversed the hierarchy from the service view perspective, let's look at the other Explore views and learn more ways to use Explore.

  15. Under Explore By, click Kubernetes Service View, (optionally you can click the drop down arrow). Four different views popup that you can select.
    Screen Shot 2019-09-04 at 10.55.17 PM.png

  16. Click Kubernetes Node View. In the Dashboard, select from the drop down, Kubernetes - Cluster Overview.

  17. The following dashboard should display. Explore is broken into two aspects. In the left pane of the Explore tab you see the hierarchy of the clusters. Every piece of infrastructure has some type of topography that explains how the components are interconnected. 


Screen Shot 2019-09-04 at 10.59.33 PM.png

On the right side of the Explore tab you can see all the Kubernetes cluster high level statistics monitoring capability, since we are in the Kubernetes - Cluster Overview dashboard.  Also notice that you are able to have panels of metrics, logs, and security all in one view. 

  1. Scroll down to the Pods Running panel. Notice that they are grouped by namespace metadata. The prod-loggen pods are not running, as indicated in red. 

    Screen Shot 2019-09-05 at 5.55.21 PM.png

  2. If you hover over the prod-loggen pods you will receive more details. 

Screen Shot 2019-09-04 at 11.07.51 PM.png

The pods that are not running are coming from prod-loggen and begin with carbonblack and pagerduty. In a Kubernetes environment the the pods are ephemeral by design for elastic ability to scale and contract. As you hover over the pods notice that in the details the pod name is given. For carbonblack service, write down the last 5 characters to the right of the hyphen. You will need this information to navigate down to the specific pod.

  1. In Explore By, click Kubernetes Node View and then Select Kubernetes Namespace View.

  2. To expand further into the hierarchy so we can see the pods in the namespace prod-loggen, click prod01.travelllogic.info and then click prod-loggen.

  3. Currently you have the Namespace Overview dashboard. To see the pod dashboard, find the carbonblack-*.* that matches the same 5 characters you wrote down back in step 24, and click carbonblack-*.-*

  4. Screen Shot 2019-09-05 at 6.18.42 PM.png

  5. The Kubernetes - Pods dashboard displays. This is how you drill down into a specific pods dashboard, allowing you to see results for Crash loop back offs (CLBOs), restarts, and last 7 days of events. You can see that the restarts are significant and that errors are showing in Pod Events, indicating that we have a problem. As you scroll down you can see both CPU and memory resource utilization, containers and at the bottom, logs together in one dashboard. Screen Shot 2019-09-05 at 6.16.30 PM.png

Quiz 

  1. The expander in the Explore By pane, changes direction to indicate if I’ve traversed down or up in the hierarchy. 

  2. Consistent metadata gives us the ability to look at our Kubernetes system from four view which are namespace, node, service and deployment.

  3. I may need to change the view in Explore By, in order to troubleshoot an event. 

Summary

Congratulations! You’ve completed these tasks in Part 2 of the Kubernetes Hands-on Labs:

  1. Applied centralized data collection to examine Kubernetes.

  2. Learned how to traverse up and down the hierarchy of a Kubernetes system using the Explore tab.

  3. Learned how to select from our four hierarchical views in the Explore tab.