The Logs-to-Metrics features allows you to extract or create metrics from log data:
- You can extract metrics that are embedded in logs. For example, your logs might contain numerical values for latency, bytes sent, request time, and so on. You can extract multiple metrics from a single log.
- You can count logs as a metric. For example, you might count the number of log messages that contain a 404 status code.
You can use the above methods in combination: you can extract metrics from logs, and also count them to create a new metric.
Once you have extracted or created metrics from your log data, you can use them like any other metrics in Sumo. You can run metric queries on them, include those queries in dashboards, and set monitors on them.
Logs-to-Metrics best practices and limitations
This section describes best practices and limitations that apply to Logs-to-Metrics rules.
No aggregations or timeslicing
Aggregation operators and timeslicing are not supported in the parse expression for a Logs-to-Metrics rule.
Manage metric cardinality
Avoid extracting metrics that contain too many unique groupings or contain dimensions that grow indefinitely, such as session IDs. For example, if you are trying to track API calls for 10,000 customers, 4 request types and 2 status codes, the metric rule could potentially create 10,000 x 4 x 2 = 80,000 unique time series (that is, 8 metrics to track per customer: count of successes/failures for each request type).
When you create a Logs-to-Metrics rule, you can test it to see how many metrics it would have created in the last 60 minutes.
Long dimension values
A logs-to-metrics rule will not create a metric dimension values longer than 250 characters. If you rule extracts a metric dimension value longer than 250 characters, the metric will not be created. A message like this will be written to the Sumo audit index:
Extracted dimension values for keys <dimensions> have more than 250 characters for logs-to-metrics rule <rule-name>
Metric volume limits and rule blacklisting
Sumo will not allow you to save a Logs-to-Metrics rule that results in more than 20,000 unique time series.
If, over time, the volume of unique time series returned by a Logs-to-Metrics rule grows to more than 100,000, Sumo will disable the rule. In that case, Sumo writes a message like this to the audit index:
User 0000000000000475 This metrics source has sent too many unique time series and has been temporarily disabled. The data sent while this source is disabled cannot be recovered. Details: type=LogsToMetrics logsToMetricsRuleId=4566, ruleName=michal_test_blacklisitng_2, ruleScope=_sourceCategory=metrics "[explainJsonPlan.metrics]" ruleParseExpression=parse "\"queryRangeEnd: \"*\"," as end, created by=Michal Math
When Sumo blacklists a rule, it sends an email notification to the creator of the rule. You can see that a rule is blacklisted on the Logs-to-Metrics page: a red icon appears to the left of the rule name.
Removing a Logs-to-Metrics rule from the blacklist
You can remove a rule from the blacklist by re-saving the rule. If you open a Logs-to-Metrics rule for editing, and save it, it will be removed from the blacklist, even if you do not make any changes to the rule. Note however, that if the rule continues to generate too many unique time series, it will be blacklisted again. So, it's best to modify your rule to make it more selective.
Field extraction rules are not supported
Logs-to-Metrics will not work with fields created from field extraction rules (FERs), since it occurs earlier in the ingestion pipeline. If you would like to use fields from an FER in your scope or parse expression, you will need to reparse those specific fields.
Logs-to-Metrics is not retroactive
Once you save a Logs-to-Metrics rule, Sumo will commence creating metrics. Sumo does not apply your rule to logs that have previously been ingested.
Supported and unsupported parsing operators
Not all Sumo parsing operators are supported. For more information see Create a Logs-to-Metrics rule.
Scheduled views and indexes are not supported
You cannot use a scheduled view or an index in the scope of a Logs-to-Metrics rule.
Enable Logs-to-Metrics rule creation by non-admin users
By default, only Sumo admins can create Logs-to-Metrics rules. To enable this feature for other users you can create a role with the Manage Logs-to-Metrics capability, or add the capability to an existing role. See Create and Manage Roles.
Create a Logs-to-Metrics rule
This section describes how to create a Logs-to-Metrics rule.
- Go to Manage Data > Metrics > Logs-to-Metrics in the Sumo web app. The page displays a list of existing Logs-to-Metrics rules.
- To create a new rule, click the plus sign (+) in the upper right of the page. The Add Logs-to-Metrics Rule page appears.
- In the Parse Log Messages section:
- Rule Name. Specify a rule name.
- Scope. Enter a query that returns the log messages from which you want to extract or calculate metrics. For best performance, enter a scope that returns only the log messages from which you want to create metrics. For example:
_sourceCategory=alert !info !warn
Once you enter a valid scope, 10 recent matching log lines appear in the Preview Parse Expression section of the page.
- Parse Expression. Enter a parse expression to extract desired fields from the logs that match the scope query. The parse operators supported in logs to metrics rules are listed below. Other parse operators are not supported.
Here is an example of a parse expression:
parse "[hostId=*]" as hostid
After you enter a valid parse expression, extracted fields appear in the Select Metrics and Dimensions section of the page. Note that in addition to any fields you extracted with your parse expression, the following Sumo metadata fields are listed:
- On the Select Metrics and Dimensions section of the page, you define the metrics and dimensions the rule will extract. The selections in the screenshot below extract one dimension from matching log messages and creates a metric from the count of matching log messages.
- Click the Metrics checkbox for a field that is a numerical value that you want to plot and analyze, such as latency. You can designate more than one field as a metric, as long as they are both numerical values. On the other hand, if your logs don’t contain embedded metrics, and you are creating a metric based on log count, you will not mark any of the fields as metrics.
- Click the Dimensions checkbox for fields by which you’d like to query and aggregate metrics, for example hostId. Configuring fields as dimensions is optional. Avoid selecting too many dimensions, as that will increase the number of unique metrics produced from the rule.
- Toggle the Count the number of log messages switch on if you want to create a metric that is the total number of log messages that match your scope and parse expression. You can create a calculated metric like this in addition to designating one or more fields as metrics. If you toggle this switch, the page prompts you to enter a name for the metric.
- Click Estimate DPM to see how many data points per minute (DPM) your rule would have created in the last 60 minutes.