With the use of out-of-the-box dashboards installed with our Kubernetes app, the Explore view gives us the ability to traverse the hierarchy and do cross signal correlation across the Kubernetes environment. In this lab, you will be familiarized with our new Dashboards (New), learn how to edit existing dashboards and search for information using metadata.
Examine an existing dashboard
Dashboard (New) is a unique framework that allows you to view logs and metrics data on the same dashboard in an integrated and seamless view. This gives you the same control over how your metrics and log data are visualized.
First let’s open an Explore tab. On the Home page, click +New.
Change your view to the deployment view, under Explore By, click Kubernetes Service View.
Select Kubernetes Deployment View.
Let’s navigate down to a particular dashboard, click prod01.travellogic.info
Notice that as you clicked on the Explore left navigation pane, that the dashboards have been changing. Let's find out why this happens.
Take a look at our Google apps deployment dashboard, click googleapps.
At the top of the dashboard. There is a grayed-out section under the Dashboard's name. You will see cluster prod01.travellogic.info, namespace prod-loggen, and deployment googleapps. This section captures and displays the path you traversed to in the hierarchy, using the left navigation pane.
These are all key value pairs (aka metadata) that have been passed to the Dashboard (New), acting as filters controlling what the dashboards displays. Our Dashboards (New) are completely customizable to whatever key value pairs you would like to use as filters. You can even trigger off of your own metadata.
Click the lower details icon at the top right of the dashboard and Select Open in Dashboards (New).
The dashboard we had just navigated to, called Kubernetes - Deployment Overview, will open in a Dashboard (New) tab. This is how you open an existing dashboard.
From here you can make edits to the dashboard. Hover (or click) in Pods Running and select the 3 details icon at the top right of the panel, click Open in Metrics.
This is the underlying query used to generate the panel. If desired, you can make edits to existing panels and save. For more information see lab Part 5: Modify your dashboard to do so.
Searching using the Structured Query Builder in Dashboards (New)
Now that you are familiar with the concept of examining a Dashboard (New), let’s use metrics to search for Kubernetes information.
- To open a new Dashboard (New), on the Home page, click + New.
- Select Dashboard from the drop-down list.
Now let’s add a time series panel. Click Add Panel and select Honeycomb.
You have the option of selecting either Logs or Metrics. To get to metrics, Click Logs and select Metrics.
Let's search for the CPU usage per node for our cluster prod01.travellogic.info using the Structured Query Builder. At the metrics window, inside the Structured Query Builder section at the top, you will see a From. In the From, enter cluster=prod01.travellogic.info. Notice the look ahead will try to find what you are looking for. This applies the Kubernetes metadata to specifically request a cluster that you will run a search upon.
Use the Structured Query Builder again, to specify the metric for CPU usage per node. In the get metric, enter node_cpu_utlisation:avg1m
In the Preview Table you will see any queries that matched to your request.
Click on the Chart tab. You will see the visual giving the pods that contained the metric within that cluster.
- What if you wanted to access create a new query builder line? At the right end of the Metrics query, click the plus sign. This will add a query builder line for you.
- You again have the option to select either Metrics or Logs for your query builder. To toggle back to Logs, Click Metrics and select Logs.
- Now you have a Logs query builder line available to modify as needed.
- What if you wanted to customize the honeycomb visual to be circles and reflect your desired thresholds? Open a new Dashboard (New) click +New. Then select honeycomb.
- You can use errors as a keyword to filter incoming logs that contain the keyword error and then aggregate by counting by source hosts using this code for the last 60 minutes and click enter to execute.
errors | count by _sourceHost
- You may change the shapes to circles, using the drop down, click Shape, then click Circle. Then select the threshold levels you prefer. Here green is anything below the value of 300, and red is above the value of 501. You may need to hover to see the values of each circle to investigate their values to choose appropriate cutoffs.
- What if you want to save this chart to a new dashboard? At the top right, click Add Dashboard.
- A Dashboard (New) tab will open displaying this chart as a panel in a new dashboard. The title of the dashboard by default it time stamped.
- You may click and change the title to <your name>lab3 and then click to save the new title. We will learn how to add more panels in the next lab.
Quiz (True or False?)
Traversing the hierarchy on the left of the Explore tab displays different dashboards which come out-of-the-box configured for Kubernetes components.
I can’t display logs, metrics, and events all in one Explore tab.
Congratulations! You’ve completed these tasks in Part 3 of the Kubernetes Hands-on Labs:
- Analyzed an existing Kubernetes dashboard.
- Using the structured query builder in Dashboard (New).
- Learning how to chart in Dashboard (New)
- Learn how to save a chart to Dashboard (New)