Skip to main content
Sumo Logic

predict

The Predict operator uses a series of time stamped numerical values to predict future values. The predict operator is useful in the following cases:

  • As an early warning system, alerting you when a threshold is about to be reached.
  • For resource and capacity planning, helpful for determining seasonal impacts, like a Cyber Monday rush on an ecommerce site.
  • Improved risk calculation.

For example, you could use this operator to take your current disk space capacity numbers, and predict when your system might run out of disk space. In these cases, the sooner an operations manager is informed that a key threshold is about to be reached the more effective he or she can plan to avoid risks and potential service degradation.   

The Predict operator now also includes support for the autoregressive (AR) model, which predicts future data points, along with the linear regression that predicts existing data point.

The AR model takes an order parameter, which is used to decide the window size for the model.

You can choose to pass an order value in the query. If not specified in the query, Sumo Logic uses order =  max(10, 10% * ts size).

If a missing data point is encountered in the generated time series, Sumo Logic uses a zero value and issues a warning to alert you. If you would like to use customized values, you can do so with the fillmissing operator.   

Syntax:

Using the default linear regression model:

  • ... | timeslice 1m | count by _timeslice | predict _count by 1m

The syntax takes the input points and outputs of two additional fields based on the original field _count:

  • _count_predicted - This is the value predicted by a simple linear model.
  • _count_error - This is the value predicted minus the actual number.

Using the more advanced autoregressive model:

  • … | timeslice 1m | count by _timeslice | predict _count by 1m model=ar

The autoregressive option (model=ar) produces the following fields in the output:

  • _count_predicted - This is the value predicted by the autoregressive algorithm.
  • _count_linear - This is the value predicted by a simple linear model.
  • _count_error - This is the value predicted by a simple linear model minus the actual number.

Optional: You can also use add the forecast parameter to control how many points into the future to predict.

  • ... | timeslice 1m | sum(processTime) by _timeslice | predict _sum by 1m forecast=5,model=ar, ar.window=10
  • ... | timeslice 1m | max(latency) by _timeslice | predict _max by 1m forecast=5m
  • ... | timeslice 1m | sum(processTime) by _timeslice | predict _sum by 1m forecast=5, model=ar
  • ... | timeslice 1m | sum(processTime) by _timeslice | predict _sum by 1m model=ar, forecast=5

The forecast parameter uses the same syntax as the timeslice operator and accepts either:

  • A simple number, such as forecast=5, which predicts five data points into the future.
  • A time granularity, such as forecast=5m, which predicts five minutes into the future.
  • If you don’t specify a value for the forecast parameter, the operator will default to three data points into the future, or forecast=3.

Rules:

  • The Predict operator must appear after a group by aggregator, such as count, min, max, or sum.
  • Use _timeslice as the key after the aggregator, for example:
    • count by _timeslice
    • max(latency) by _timeslice
    • avg(numLogins) by _timeslice

Limitations:

These internal limitations are meant to provide "speed bumps" to ensure best performance.

  • Predict will not use more than 10,000 input points to estimate the model.
  • Predict will not forecast more than 100 points into the future.
  • Predict will not interpolate more than 20,000 input points. (Predict adds "phantom" input points where there should be a timeslice, but no data point is present.)

Advanced options for Autoregressive Model

If there are cyclical patterns that fit within the ar.window, the autoregressive algorithm will learn the cyclical pattern and uses that in prediction.

For example, if there is an hourly cyclical pattern, the following query will learn that cycle:

… | timeslice 5m
| <aggregate function> by _timeslice as _val  
| predict _val by 5m model=ar, ar.window=15

In this query, the window size (15 consecutive data points) covers more than 1 hour (15 data points * 5m interval = 75 minutes). So if there are cyclical patterns with a period of less than 75 minutes, the model will discover them.

In the following screenshot, you can see the training data in light blue, and the predicted data in dark blue.

Then in the next screenshot, you see the actual data, and how closely that matches the prediction.

Autoregressive Example

For example, the following query counts the number of messages that contain an error term for every half minute:

_sourceCategory=taskmanager jobState=InQueue error
| timeslice 30s
| count by _timeslice

The original query generates a time series and based on it, you can construct a prediction query such as:

_sourceCategory=taskmanager jobState=InQueue error
| timeslice 30s
| count by _timeslice
| predict _count by 30s model=ar,ar.window=50,forecast=100

This version uses the autoregressive model to predict 100 data points in the future based on the existing data points.  

Best Practices for the AR Model

Data Points

Because the autoregressive model is an advanced option, it requires more data points than the regular linear regression option to train a proper model. We recommended that you generate more than 100 data points for the AR model. If the number of data points is less than 100, Sumo Logic displays a warning stating that the learned model may not be optimal.

Window Size

The window size greatly impacts the performance of the autoregressive model. As a result, Sumo Logic automatically chooses the best default value for your query.

If you want to customize the window size, we recommend that you to set the window size to be larger than 40% of the data points in the time series. However, don’t make the window size too large. The maximum would be 60%. Otherwise, the model will not be trained properly.

Examples

Predict events in a Source Category.

In this example, we are logging jobs in a queue, and we want to project when the queue will be empty.

_sourceCategory=taskmanager
| jobState=InQueue
| timeslice 1m
| count by _timeslice
| toDouble(_count)
| predict _count by 1m forecast=5

The query returns an aggregation table with columns for _count, _count_predicted, and _count_error:

From here, you can select the Line Chart icon, and automatically create a Combo Chart that represents the _count_error as a column chart, and the _count and _count_predicted mapped on top of that with separate lines. The _count_predicted value will tell you when the value will reach zero.

Monitoring max response time from IIS logs.

This query searched for IIS log response times and predicts the maximum response time from the values found in the logs.

_sourceCategory=IIS/Access
| parse regex "\d+-\d+-\d+ \d+:\d+:\d+ (?<server_ip>\S+) (?<method>\S+) (?<cs_uri_stem>/\S+?) \S+ \d+ (?<user>\S+) (?<client_ip>[\.\d]+) "
| parse regex "\d+ \d+ \d+ (?<response_time>\d+)$"
| timeslice 15m | max(response_time) as response_time by _timeslice | predict response_time by 15m forecast=60m

It returns an aggregation table. Select the Line Chart icon, and click the absolute value. This automatically create a Combo Chart that visually graphs the predicted results.

Predict IIS max response time

Monitoring data received by a specific load balancer.

This example looks at data received by a specific load balancer in AWS Elastic Load Balancing logs.

_sourceCategory=aws_elb2 long-api-lb
| parse "* * *:* *:* * * * * * * * \"* *://*:*/* HTTP" as f1, elb_server, clientIP, port, backend, backend_port, requestProc, ba_Response, cli_Response, ELB_StatusCode, be_StatusCode, rcvd, send, method, protocol, domain, server_port, path
| timeslice 30m
| (rcvd/(1024)) as mbytes1 | sum(mbytes1) as data_received by _timeslice
| predict data_received by 30m forecast=120m

It returns an aggregation table. Select the Line Chart icon, and automatically create a Combo Chart that visually graphs the predicted results.

Use predict output values to create a Scheduled Search Alert

You can set filtering criteria for the predict output values and create a Scheduled Search alert based on the results, which will notify you when the outputs surpass a certain value.

For example, if you started with this original query:

_sourceCategory=taskmanager jobState=InQueue
| timeslice 30s
| count by _timeslice
| predict _count by 30s model=ar,ar.window=50,forecast=100,

the outputs would be _count, _count_predicted, _count_error, and _count_linear. The query would produce a line chart that looked like this:

When you select the table chart icon, you can see the results like this. Notice there are no results in the _count_predicted column for the original data points, which are only applicable to the predicted data.

If you wanted to know how many _count_predicted are above 8,000, you could write a new query like this:

_sourceCategory=taskmanager jobState=InQueue
| timeslice 30s
| count by _timeslice
| predict _count by 30s model=ar,ar.window=50,forecast=100
| where !isNull(_count_predicted)
| where _count_predicted > 8000

Then when you run the search and select the table chart icon, you would see results such as:

Based on the result, you can create a Scheduled Search email alert to notify you when there are too many _count_predicted above the threshold of 8,000.