Skip to main content
Sumo Logic

Lab 8 - Relating metrics to logs by using Key Value pairs and advance comparison operators

  1. Last updated
  2. Save as PDF
This lab shows an example of how to use business and operational from a fictitious travel application's data to populate metrics.
You can also use metrics as indicators for troubleshooting by connecting them with related logs. These metrics were generated using metric rules, which parses metrics from logs. We will also look at a couple of metric operators for comparison. In this lab we will connect our logs with the metrics and then also explore advance metric operators timeshift, delta, and rate of change.

NOTE: We will be using Advanced Mode, which is shown in  Lab 6


Key Value Pair usage

In this lab, we will look at metrics from our TravelLogic Demo. These metrics were generated using metric rules, which parses metrics from logs. We can go look at these metric rules click Manage Data>Metrics>Metrics Rules.

Let's begin by testing the Graphite Metrics Rule we just reviewed during Training.

  1. First, let's query a metric using its raw Graphite name. In a metrics query, enter the following:

    travel.training.counters.travel-checkout*.bookings.success.count

  2.  On a second query, run the following to query the exact same metric. However, this time, we are taking advantage of the key-value tags created by the Metrics Rule

    type=bookings metric=success.count
    Remember to select Advanced Mode on the second query when you create it.

    clipboard_eef534e1e7982cad1116e912816daed88.png

  3. You will notice you are graphing the exact same metric twice.
    clipboard_e96f563d221b9d7dde8674fae36ab0a2a.png
    You can check this by clicking on the items listed in the lower left corner to remove their traces.

Now let's learn how to use some additional operators. In the next steps we'll plot data from an online travel website to determine successful versus unsuccessful bookings.

Travel Logic Demo Usage  

  1. In a new Metrics tab, add a query to search for all your successful bookings for the last 60 minutes:

    type=bookings metric=success.count

  2. In a second query underneath the first one, search for all failed bookings:

    type=bookings metric=fail.count

    clipboard_ece6d07a6f2b5f0a28a092eb093b72fa5.png

  3. Click on the Chart tab to see what options are available to you. For example, you can change the chart type (line, pie, single value or honeycomb), the color palette used, the line width and the axes labels and scales.
    clipboard_e2b2d0d9281a7ff5a0cc9bcd258312208.png

  4. Explore the Display Options tab, which is located on the far right edge.  This allows you to change the Display, Display overrides, Axes, Legend and JSON values that make up this chart.
    clipboard_e16404b6440fbceacd90eeca21124504d.png

  5. Back in the query tab, toggle off the success.count by clicking on the metric=success.count _sourceHost=100.126.237.166 on the bottom left corner. This will now only chart the fail.count metric.
    clipboard_e2c839c42823266044815a84b778c6f4e.png
     

Lastly, click on the 3 grey dots in the top right corner to view the query info, refresh the query, or add this chart to a Dashboard.

Lastly, let's learn how to correlate metrics to relevant logs to identify the root cause.

Metrics allow you to identify symptoms in your environment (WHAT is going on?). Relevant logs help you identify the cause (WHY is this happening?). Let's again look for successful and failed bookings, but this time, let's take a look at the relevant logs to identify why we have failed bookings.

  1. Identify counts of successful booking and failed bookings for your travel website.

    clipboard_ece6d07a6f2b5f0a28a092eb093b72fa5.png

  2. To overlay your metrics with the relevant logs, enter this log query as depicted below:

    _sourceCategory=*training/travel/checkout* error | timeslice 1h | count by _timeslice

    clipboard_e96615ca68766abf17e5b43bc75dd870e.png

  3. Change the time to -24h as shown below
    clipboard_e13cad124c771dab5794f0597c3e1d9fa.png
    You will notice the the count line is shown, but due to the scale the other lines are suppressed at the bottom on the chart.  Lets fix this

  4. Select the Display overrides tab on the far right side and select the #C as the Query or series name.  Under Style select AxisYType and set it to Secondary. 
    clipboard_e3d09b5de152cdb35da3fb9010389a850.png
    Your result should now look like this
    clipboard_e3d12c5ffde6a733718bc294fd55d3715.png

Timeshift Operator


Let's now compare KPIs at different time periods using the timeshift operator. The timeshift operator shifts the time series of your query. It's very useful to compare across multiple time periods.

  1. In a new Metrics tab, add a query to search for all your mean latency for the last 60 minutes.

    metric=latency.mean

  2. Compare that with your latency from 1 day ago.

    metric=latency.mean | timeshift 1d


clipboard_eb436389202807acd9dd94e70b9d12c29.png
Similar to logs, metrics have  the usual operators (min, max, sum, count, avg). However, oftentimes, what you want to measure is change.

Rate of Change Operator


In this next exercise, we will identify rate of change to get early warning on impending issues.

  1. In a new Metrics tab, add a query to search for a count of packets received in the last 60 minutes.

    type=packets_received metric=count

  2. To find the difference between one data point and the next, edit your query to show the delta.

    type=packets_received metric=count | delta

  3. However, to find the rate of change, in this case, packets received per second, edit your query to

    type=packets_received metric=count | rate

With this last query, you're able to determine if the rate at which packets are being received is increasing gradually or spiking quickly. Identifying an outlier on a rate of change is a better indicator of an impending problem.

clipboard_e594fdb5f63e69782165bbcbe796497c9.png