Skip to main content
Sumo Logic

Monitors

Sumo Logic Monitors allow you to set robust and configurable alerting policies that enable you to get notified about critical changes or issues affecting your production application.

Monitors track your Metrics or Logs data in real time and send notifications when noteworthy changes happen in your production applications.

Rules

  • You need the Manage or View Monitors role capability to manage or view Monitors respectively.
  • Your account can have up to 300 Log monitors and 500 Metrics monitors.
  • The frequency a Monitor executes depends upon a variety of factors such as the underlying query, the operators used, and the detection window. It can vary from a couple of seconds to a few minutes. If for example, the detection window of your alert is one day it will be evaluated every couple of minutes, whereas if the detection window of the monitor is 15 minutes then it will be evaluated every couple of seconds.
  • Log Monitors use the role search filter of its creator.
  • Log Monitors delay execution by two minutes. This means it won't evaluate data from the current time, but evaluate data from two minutes ago.  This ensures that any delays in ingestion are factored in and won't generate false positive or false negative alerts.

Limitations

  • Receipt Time is not supported.
  • An aggregate Metrics Monitor can evaluate up to 15,000 time series. A non-aggregate Metrics Monitors can evaluate up to 3,000 time series.
  • Save to Index and Save to Lookup are not supported.
  • Search templates are not supported.
  • After one day if there is no update to an incident, the system automatically expires it. The incident is marked as resolved with the resolution set to Expired.
  • A Logs Monitor can have one query up to 4,000 characters long. Metrics Monitors can specify up to six queries.
  • Dynamic Parsing (auto-parse mode) is not supported.
  • The timeshift metrics operator is not supported in a Metrics Monitor.
  • Hidden Metrics queries do not persist across edit sessions.
  • The last millisecond of the defined time range is not searched. For example, a time range of 6:15 to 6.30 pm will run as 6:15:00:000 to 6:29:59:999.

Notifications

  • Monitors keep track of the notifications it sends and won't send additional notifications after sending the first one if the incident persists. It will only send additional notifications if there is a major change in the state of the monitor, such as new triggers from Warning to Critical or a different time series is detected.
  • Auto Resolution is currently supported with Email, Microsoft Teams, OpsGenie, PagerDuty, and Slack connections. Support for other connection types is coming soon.
  • When more than one trigger condition is met notifications are sent based on the trigger conditions:
    • When both Critical & Warning conditions are met two separate notifications are generated, one for the Critical condition and one for the Warning condition. Auto resolution, if set up, will work according to the resolution condition for each case. 
    • When the Missing Data condition is met after initially firing a critical/warning incident the system will resolve the Critical or Warning incident that was created with the appropriate resolution reason. The system will also create a new Missing Data incident and notify you appropriately using your configured notification channel.
  • Metrics Monitors support group notifications. When configured, the monitor will not trigger new notifications until the first one is resolved. The monitor will only update if the notification type supports Auto Resolution. Grouped notifications will resolve when all the time series return to normal.
  • The Recovery is evaluated after the detection window duration of the triggered alert has passed. For example, if an alert is triggered at 1 PM and the detection window was 15 minutes, the alert will only start evaluating the recovery condition after 1:15 PM. In other words, the alert will not resolve before 1:15 PM in any case. This is to ensure there isn't an alert between the triggered and resolved state, especially for metrics that are volatile.

Monitors page

The Monitors page allows you to view, create, and manage Monitors. You can create and organize your Monitors into folders. To access the page go to Manage Data > Alerts > Monitors.

monitors page april 2021.png

The page displays the following information:

  • Name. Name of the monitor.
  • Status. A monitor is considered Normal when none of the trigger conditions are met and your data is actively being monitored.
  • Type. A Monitor type is either Logs or Metrics, based on the type of data being monitored.
  • Query. The query the monitor is running to monitor your data.
  • Created By. The user that created the monitor.
  • Last Modified. The time the monitor was last updated.
  • Capacity Used. The bottom of the table shows the number of Logs and Metrics Monitors in your account.

At the top of the page, you can:

  • Search Monitors. Use the search field to filter monitors by name and status. For example, you can view all monitors that are currently triggered in the system by clicking the Status: All Triggered.
    seach monitors input.png
  • Click Add to:
    Add monitors page.png
    • create folders for organizing your monitors.
    • create a new monitor.
    • import monitors from the exported JSON you copied from the More Actions menu in the Details pane of the original monitor.

Details pane

The details pane provides additional information about a selected monitor, like its query, trigger conditions, and notification preferences. For the monitors listed, select a row to view its details. A details pane appears to the right of the table.

sumo logic monitor.png

In the details pane you can see the following details for a monitor:

  • Name of the monitor.
  • The status of the monitor, either Normal, Critical, Warning, or Missing Data. A monitor can be in multiple states at the same time.
  • Description of the monitor.
  • Type of monitor, either Logs or Metrics.
  • Path is the Library location where the monitor is stored.
  • Query the monitor is running to monitor your data. 
  • Trigger Conditions that are set on the monitor. Applicable values include Critical, Warning, and Missing Data.
  • Notifications configured on the monitor.
  • The timestamp and user that Created and last Modified the monitor.

Under the More Actions menu you can:

  • Disable and Enable. A monitor that is in a disabled state will not fire any notifications.
  • Copy Path. Copy the path of the monitor to your computer clipboard.
  • Duplicate. Make another monitor based on the same settings.
  • Move. Move the monitor to a different path.
  • Export. Provides JSON of the monitor, allowing you to transfer content within Sumo Logic by copying this JSON, then pasting it into the import dialog in the Library location you choose. This JSON format may change without notice. 
  • Delete.

details more actions.png

Add a monitor

On the Monitors page, click on the Add button then New Monitor to add a new monitor. The monitor creation dialog box will appear.

  1. Select a Monitor Type, either Logs or Metrics.
    monitor type selection.png

  2. Provide a Query. A Logs Monitor can have one query up to 4,000 characters long. Metrics Monitors can specify up to six queries. When providing multiple metrics queries use the letter labels to reference a query row, see joined metrics queries for details. The monitor will automatically deduce the query row to use for the trigger. 

  3. Specify the Trigger Type. A monitor can have one critical, warning, and missing data trigger condition, each with one or more notification destinations.

    Triggers have different options depending on the query and alert type. Click the Expand button next to the query type you're using for configuration details.

Logs query

Logs query.png

Trigger Type: Critical and Warning

logs trigger type 2021.png

Alert when returned row count is <threshold type> <threshold> for <time range>

Parameter Description
Threshold type How you want the value compared. Select either greater than, greater than or equal, less than or equal, or less than.
Threshold The value against which the trigger will be evaluated. You can specify any valid numeric value up to 1,000.
Time range The duration of time to evaluate. Select either 5 minutes, 10 minutes, 15 minutes, 30 minutes, 1 hour, 6 hours, 12 hours, or 24 hours.

Recover

Use the Infer recovery settings option to set the recovery to the opposite of the alert setting. The Alert must be checked for the option to be available.
logs trigger recovery toggle.png
For example, when the alert is set to > 10 the recovery would be set to <= 10 when inferred.

Sumo Logic automatically resolves the incident when the resolution condition is satisfied.

Recover automatically when result is <threshold type> <threshold>

Parameter Description
Threshold type How you want the value compared. Select either greater than, greater than or equal, less than or equal, or less than.
Threshold The value against which the resolution will be evaluated. You can specify any valid numeric value.


        Trigger Type: Missing Data
        
       logs missing data Jan 2021.png

Alert when missing data within <time range>

Parameter Description
Time range The time span of data to evaluate. Select either 5 minutes, 10 minutes, 15 minutes, 30 minutes, 1 hour, 6 hours, 12 hours, or 24 hours.

Recover

  • Automatically: Sumo Logic automatically resolves the incident when the resolution condition is satisfied. Recover automatically when data becomes available for the affected time span.

Metrics query

metrics query.png

Trigger Type: Critical and Warning

metrics trigger types.png

Alert when result is <threshold type> <threshold> <occurrence type> <time range>

Parameter Description
Threshold type How you want the value compared. Select either greater than, greater than or equal, less than or equal, or less than.
Threshold The value against which the trigger will be evaluated. You can specify any valid numeric value.
Occurrence type The time condition you want for the trigger. Select either at any time within or continuously for

Choose continuously for if you want all the data points for the given metric to meet threshold conditions in a given time range, before triggering an alert. Alternatively, choose at any time within if you want to generate an alert when only a single data point meets the threshold condition for the given time range.
Time range The duration of time to evaluate. Select either 5 minutes, 10 minutes, 15 minutes, 30 minutes, or 1 hour.

Recover

Use the Infer recovery settings option to set the recovery to the opposite of the alert setting. The Alert must be checked for the option to be available.
metrics trigger recovery toggle.png
For example, when the alert is set to > 10 the recovery would be set to <= 10 when inferred.

Sumo Logic automatically resolves the incident when the resolution condition is satisfied.

Recover automatically when result is <threshold type> <threshold>

Parameter Description
Threshold type How you want the value compared. Select either greater than, greater than or equal, less than or equal, or less than.
Threshold The value against which the resolution will be evaluated. You can specify any valid numeric value.

Trigger Type: Missing Data

missing.png

Alert when missing data <occurrence type> for <time range>

Parameter Description
Occurrence type The time condition you want for the trigger. Choose either for all or for any.

If you choose all you will get notified when all of the metrics meeting the query condition are not sending data in the given time range.

Alternatively, you can choose any if you want to get notified when one of the metrics does not receive any data in the given time range. This option requires at least one initial data point and expires after 24 hours once triggered.
Time range The duration of time to evaluate. Select either 5 minutes, 10 minutes, 15 minutes, 30 minutes, or 1 hour.

Recover

  • Automatically: Sumo Logic automatically resolves the incident when the resolution condition is satisfied. Recover automatically when data becomes available for the affected time span.
  1. (Optional) Set Notifications, when a trigger condition is met you can send notifications to other people and services. To add notifications click on the Add Notification button. You can add more than one notification channel for a monitor.

    notifications input pane.png

    Metrics Monitors have an option to send notifications either as a group or separately. Group Notifications define whether you want single notifications per time series that match the monitor query or you want group notifications where you receive a single notification for the entire monitor.
    1. The Connection Type specifies the notification channel where you want to get notified, such as an email or webhook. See Connections for details. 

      Monitor notifications support variables to reference monitor configuration settings or your raw data. See alert variables for a table of the available variables.

      • Email: You must provide one or more recipient email addresses. You can customize the email Subject and Body.

      • Webhook: By default, the payload defined on the Connection is used. You can customize your payload for each notification if needed.

    2. Select the Alert and Recovery checkboxes for each trigger type based on when you want to send a notification.  You can have different Trigger Conditions send a notification to different channels. For example, you can get notified on PagerDuty for Critical Incidents and get an email or slack notification for Warning incidents.

      If your connection type is Microsoft Teams, OpsGenie, PagerDuty, or Slack the Recovery checkbox enables an automatic resolution process that updates the connection when an alert has recovered within Sumo Logic. Support for other connection types is coming soon.
    3. Add Notifications to add additional notification channels as needed. You can configure different notifications for each trigger type, critical, warning, and missing data.

  2. Enter a Name for the monitor and the Location you want it saved to. A Description is optional.

    monitor details during creation.png

Disable and enable a monitor

A monitor that is in a disabled state will not fire any notifications.

  1. Find and select the monitor in the Monitors table. A three-dot icon appears on the right of the row.
    menu option.png
  2. Click the three-dot icon to view the menu options. You can select to Enable or Disable the monitor.