Lab 8 - Relating metrics to logs by using Key Value pairs and advance comparison operators
You can also use metrics as indicators for troubleshooting by connecting them with related logs. These metrics were generated using metric rules, which parses metrics from logs. We will also look at a couple of metric operators for comparison. In this lab we will connect our logs with the metrics and then also explore advance metric operators timeshift, delta, and rate of change.
NOTE: We will be using Advanced Mode, which is shown in Lab 6
Key Value Pair usage
In this lab, we will look at metrics from our TravelLogic Demo. These metrics were generated using metric rules, which parses metrics from logs. We can go look at these metric rules click Manage Data>Metrics>Metrics Rules.
Let's begin by testing the Graphite Metrics Rule we just reviewed during Training.
-
First, let's query a metric using its raw Graphite name. In a metrics query, enter the following:
travel.training.counters.travel-checkout*.bookings.success.count -
On a second query, run the following to query the exact same metric. However, this time, we are taking advantage of the key-value tags created by the Metrics Rule
type=bookings metric=success.count
Remember to select Advanced Mode on the second query when you create it.
-
You will notice you are graphing the exact same metric twice.
You can check this by clicking on the items listed in the lower left corner to remove their traces.
Now let's learn how to use some additional operators. In the next steps we'll plot data from an online travel website to determine successful versus unsuccessful bookings.
Travel Logic Demo Usage
-
In a new Metrics tab, add a query to search for all your successful bookings for the last 60 minutes:
type=bookings metric=success.count -
In a second query underneath the first one, search for all failed bookings:
type=bookings metric=fail.count
-
Click on the Chart tab to see what options are available to you. For example, you can change the chart type (line, pie, single value or honeycomb), the color palette used, the line width and the axes labels and scales.
-
Explore the Display Options tab, which is located on the far right edge. This allows you to change the Display, Display overrides, Axes, Legend and JSON values that make up this chart.
-
Back in the query tab, toggle off the success.count by clicking on the metric=success.count _sourceHost=100.126.237.166 on the bottom left corner. This will now only chart the fail.count metric.
Lastly, click on the 3 grey dots in the top right corner to view the query info, refresh the query, or add this chart to a Dashboard.
Lastly, let's learn how to correlate metrics to relevant logs to identify the root cause.
Metrics allow you to identify symptoms in your environment (WHAT is going on?). Relevant logs help you identify the cause (WHY is this happening?). Let's again look for successful and failed bookings, but this time, let's take a look at the relevant logs to identify why we have failed bookings.
-
Identify counts of successful booking and failed bookings for your travel website.
-
To overlay your metrics with the relevant logs, enter this log query as depicted below:
_sourceCategory=*training/travel/checkout* error | timeslice 1h | count by _timeslice
-
Change the time to -24h as shown below
You will notice the the count line is shown, but due to the scale the other lines are suppressed at the bottom on the chart. Lets fix this -
Select the Display overrides tab on the far right side and select the #C as the Query or series name. Under Style select AxisYType and set it to Secondary.
Your result should now look like this
Timeshift Operator
Let's now compare KPIs at different time periods using the timeshift operator. The timeshift operator shifts the time series of your query. It's very useful to compare across multiple time periods.
-
In a new Metrics tab, add a query to search for all your mean latency for the last 60 minutes.
metric=latency.mean -
Compare that with your latency from 1 day ago.
metric=latency.mean | timeshift 1d
Similar to logs, metrics have the usual operators (min, max, sum, count, avg). However, oftentimes, what you want to measure is change.
Rate of Change Operator
In this next exercise, we will identify rate of change to get early warning on impending issues.
-
In a new Metrics tab, add a query to search for a count of packets received in the last 60 minutes.
type=packets_received metric=count -
To find the difference between one data point and the next, edit your query to show the delta.
type=packets_received metric=count | delta -
However, to find the rate of change, in this case, packets received per second, edit your query to
type=packets_received metric=count | rate
With this last query, you're able to determine if the rate at which packets are being received is increasing gradually or spiking quickly. Identifying an outlier on a rate of change is a better indicator of an impending problem.